Crawler

March 20, 2005 Comments

Robert Clough

Robert Clough

Articles



Definition: An automated robot program that follows links to visit web sites on behalf of search engines or directories. Crawlers then process and index the code and content of a web page to be stored in the search engine's database.

Examples: Googlebot is the crawler that travels the web finding and indexing pages for the Google search engine.





About the Author

Search Engine Marketing Columnist

Search Engine Marketing Columnist