From: Jeff Laurie

Jill --

Congratulations on the spectacular success you've achieved in the SEO world.

I understand that spiders only go to the second level of a site. Unfortunately, all of our 413 previous newsletter issues are on the third level of the site. If I put a page on the second level -- an archive with links to the individual issues -- would spiders follow and index the links to the third level?

Thanks a lot in advance. Keep up the great work.

Jeff

 

~~~Jill's Reply~~~

Hi Jeff,

Actually, the spiders will go to any depth of a site, not just the second level. However, they don't place as much emphasis on pages within the site that take more than a couple of clicks to get to from the main page. It's not the physical directory structure that slows the spiders down, but really the number of clicks it takes to get to the pages.

In other words, say you have a page that's four-levels deep in the directory structure like this:

yoursite.com/dir1/dir2/dir3/dir4/deep-page.htm

Let's also say that there's no link to that page from the home page. Let's go even further and say there's no link to that page from the next level of pages (which would be common for a page buried four-directory levels deep). Let's say that people don't generally find that page (when starting at the home page) until they've clicked through from three other levels of the site. That page is pretty hard to find!

It's the same for the search engine spiders. Very often they take a really fast crawl through your site and don't bother to fetch pages that are hard to find. Sure, they may get to those pages eventually when they're in the mood for a deep crawl, but it won't happen very often.

Now let's take that same page and put a link to it from the home page of the site. It's still physically four directory levels deep, but it only takes one click for a person to find it. With that scenario, the search engine spiders will also be able to easily find it!

Now, there are of course exceptions to this rule. If other sites happen to link to your deep page from one of their top-level pages, it can have a similar effect as if you did. But you can't control how others link to you, so don't leave it up to chance! If you have a deep page that changes often, and you want to ensure that the latest version is apt to be spidered and indexed, then make it easy to find.

Your best bet to ensure all your content gets spidered is to create a sitemap that lists all your pages, and then link it visibly and directly from the home page of your site.

Hope this helps!

Jill
April 10, 2003





CEO and founder of High Rankings®, Jill Whalen has been performing search engine optimization since 1995 and is the host of the free High Rankings Advisor search engine marketing newsletter, author of "The Nitty-gritty of Writing for the Search Engines" and founder/administrator of the popular High Rankings Search Engine Optimization Forum. In 2006, Jill co-founded SEMNE, a local search engine marketing networking organization for people and companies in New England.

High Rankings is an internationally recognized search engine optimization firm located in Framingham, MA specializing in search engine optimization, SEO consultations, in-house training, site audit reports, search marketing seminars and workshops. High Rankings has a 100% success rate for substantially improving client rankings and targeted traffic.

Jill speaks at national and international conferences and has been writing about SEO and search marketing since 2000. She's been quoted in such publications as The Wall Street Journal, U.S. News & World Report and The Washington Post. Her articles have appeared in numerous print magazines and online websites including CIO Magazine, CMS Focus, The Internet Marketing Report, ClickZ, WorkZ, Inc.com, Entrepreneur, Lycos Small Business, WebProNews, SiteProNews and others. Jill has also appeared on many online and offline radio programs such as Entrepreneur Magazine's E-Biz Radio Show, SearchEngineRadio and the eMarketing Talkshow.





Search Engine Guide > Jill Whalen > To What Depth Will Search Engine Spiders Go?