While learning about search engine marketing is handy, understanding how the search engines themselves work can go a long way toward helping you get a solid grasp on improving your rankings.

To that end, Rob Sullivan has put together a pretty good article on how search engine crawlers work called "Anatomy of a Search Engine Crawler" over at Search Engine Journal.

It's easy to understand and gives a nice overview of things.

Generally, the crawler gets a list of URL’s to visit and store. The crawler doesn’t rank the pages, it only goes out and gets copies which it stores, or forwards to the search engine to later index and rank according to various aspects.

and

If your site goes down temporarily when a crawler visits repeatedly like this, don’t worry. The crawlers are smart enough to leave and come back later and try again. If, however, the continue to find the site down, or slow to respond, they may opt to stay away for longer periods, or index the site more slowly. This can negatively impact your site’s performance in the search engines.