Jill --

Congratulations on the spectacular success you've achieved in the SEO world.

I understand that spiders only go to the second level of a site. Unfortunately, all of our 413 previous newsletter issues are on the third level of the site. If I put a page on the second level -- an archive with links to the individual issues -- would spiders follow and index the links to the third level?

Thanks a lot in advance. Keep up the great work.

Jeff

Hi Jeff,

Actually, the spiders will go to any depth of a site, not just the second level. However, they don't place as much emphasis on pages within the site that take more than a couple of clicks to get to from the main page. It's not the physical directory structure that slows the spiders down, but really the number of clicks it takes to get to the pages.

In other words, say you have a page that's four-levels deep in the directory structure like this:

yoursite.com/dir1/dir2/dir3/dir4/deep-page.htm

Let's also say that there's no link to that page from the home page. Let's go even further and say there's no link to that page from the next level of pages (which would be common for a page buried four-directory levels deep). Let's say that people don't generally find that page (when starting at the home page) until they've clicked through from three other levels of the site. That page is pretty hard to find!

It's the same for the search engine spiders. Very often they take a really fast crawl through your site and don't bother to fetch pages that are hard to find. Sure, they may get to those pages eventually when they're in the mood for a deep crawl, but it won't happen very often.

Now let's take that same page and put a link to it from the home page of the site. It's still physically four directory levels deep, but it only takes one click for a person to find it. With that scenario, the search engine spiders will also be able to easily find it!

Now, there are of course exceptions to this rule. If other sites happen to link to your deep page from one of their top-level pages, it can have a similar effect as if you did. But you can't control how others link to you, so don't leave it up to chance! If you have a deep page that changes often, and you want to ensure that the latest version is apt to be spidered and indexed, then make it easy to find.

Your best bet to ensure all your content gets spidered is to create a sitemap that lists all your pages, and then link it visibly and directly from the home page of your site.

Hope this helps!

Jill