Robots.txt

March 21, 2005 Comments

Robert Clough

Robert Clough

Articles



Definition: A text file that is stored in the top-level directory of a web site to be accessed by robots or spiders that might visit the site. Robots that comply with the "Robots Exclusion Standard" will read the commands in this file and will obey them.

The primary purpose of the robots.txt file is to direct spiders to ignore directories that may contain private or unnecessary information.

Examples: The example below attempts to prevent all robots from visiting the /test files directory:

User-agent: *
Disallow: /testfiles





About the Author

Search Engine Marketing Columnist

Search Engine Marketing Columnist