Robots exclusion standard

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is unrelated to, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

Other News:

Images »

Videos »

  • Web Design Blog - Robots txt files explained

    Web Design Blog - Robots txt files explained

    Nick from the Creare Group explain the use of a robots files to restrict search engine bots for individual files, folders and database queries.
  • KeywordEnvy Tutorial #1: robots.txt

    KeywordEnvy Tutorial #1: robots.txt

    This week we show how to use your robots.txt file to prevent Google from indexing querystring params in their results. Use KeywordEnvy at www.keywordenvy.com
  • add sitemap tag to robots.txt

    add sitemap tag to robots.txt

    www.howtodowebmarketing.com How to add a sitemap to the robots.txt file for your website. This is important as its the only way of defining your sitemap to yahoo and Bing will complain if this line is not present within your websites robots.txt file. It goes without saying that unless your sitemap is found by the search engines your pages will struggle to be seen when people search.
  • Online Marketing Quick Tip #1 - Search Engine Optimization - Robots.txt files

    Online Marketing Quick Tip #1 - Search Engine Optimization - Robots.txt files

    The first installment of weekly online marketing quick tips hosted by Mike Rynchek of Spyder Trap Online Marketing. To learn more about Online Marketing Quick Tips, or about Spyder Trap Online Marketing, visit www.spydertrap.com
  • Using Robots.TXT ad sprider files for SEO

    Using Robots.TXT ad sprider files for SEO

    Chris reviewing creating a robots.txt file with Lytico.com
  • Use Google Webmaster Tools to Create a robots.txt File

    Use Google Webmaster Tools to Create a robots.txt File

    I demonstrate how to use the Google Webmaster Tools to create a robots.txt file designed to protect sensitive areas of your blog or website.
  • Should I block duplicate pages using robots.txt?

    Should I block duplicate pages using robots.txt?

    Halfdeck from Davis, CA asks: "If Google crawls 1000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using META ROBOTS NOINDEX,NOFOLLOW a better alternative?" Short answer: No, don't block them using robots.txt. Learn more about duplicate content here: www.google.com
  • Uncrawled URLs in search results

    Uncrawled URLs in search results

    Matt Cutts explains why a page that is disallowed in robots.txt may still appear in Google's search results.

  • Web Design Blog - Robots txt files explained

  • KeywordEnvy Tutorial #1: robots.txt

  • add sitemap tag to robots.txt

  • Online Marketing Quick Tip #1 - Search Engine Optimization - Robots.txt files

  • Using Robots.TXT ad sprider files for SEO

  • Use Google Webmaster Tools to Create a robots.txt File

  • Should I block duplicate pages using robots.txt?

  • Uncrawled URLs in search results

Related Sections

©2010 Copyright Daymade - Privacy Policy