From: Jason Abbate

Jill -

We are likely to be launching some sites in the near future that will be fully dynamic, with all of the content contained in a database. We want to make sure that we make the right SEO decisions. As far as I understand, using direct submissions, we can have the pages indexed, but we are concerned that spiders will not be able to read this content. We have looked into XML feeds but the minimum is 500 urls and the sites we're doing are only 40-60 pages.

Is there any other approach we can take to ensure that the content is spidered?

Thanks for your thoughts.

Jason Abbate



~~~Jill's Response~~~

Hi Jason,

You are not alone in wondering how you can ensure that the search engines index your dynamic site pages. This is the question I get asked most often these days. Since this is such a prevalent issue, I'm going to republish the interview I had with my friend and colleague, Alan Perkins. It's been just about a year since I interviewed Alan, but the information is as good today as it was in 2002.

Alan has been working with search engines since 1995. He holds patents in search engine technology and was lead developer of Search Mechanics, a product to help webmasters make their sites more search engine friendly. Alan is also the co-founder of e-Brand Management, a company dedicated to helping people build and maintain a successful online presence.

So without further ado, here's my interview with Alan:

Jill: Can you explain what dynamic content and dynamic URLs are?

Alan: The terms dynamic URL and dynamic content are frequently used interchangeably. However, this can lead to confusion because they are two separate, but related, terms. A URL is not content - a URL is the address of some content.

Dynamic content is information that is delivered to the Web browser in a different form than it exists on the server. It is usually pulled from a database and created on the fly at the server level through CGI programming, ASP, PHP, or by a content management system such as BroadVision(tm) or ATG Dynamo(tm).

Dynamic URLs, on the other hand, are simply Web site addresses that contain a question mark (?).

In contrast, static content is stored on the Web server in the same format that is delivered to the Web browser. And static URLs do not contain question marks.

In general, dynamic URLs are addresses of dynamic content, and static URLs are addresses of static content. However, this need not be the case, as we shall see later.

Jill: We often hear that search engines have a problem indexing dynamic content; why is this?

Alan: It boils down to two issues -- the same core content seen at different URLs, and different core content seen at the same URL.

When the same core content is at different URLs, a small site can appear to be very large because an unlimited number of URLs can be used to provide essentially the same content. Spiders can fall into "dynamic spider traps," crawling through thousands of URLs when only a few really needed to be crawled. Since a dynamic URL usually indicates dynamic content, the simplest way for a search engine to avoid these spider traps is to avoid dynamic URLs altogether. Remember, search engines want to index any given core content just once.

Now let's consider different core content at the same URL. There are a number of ways in which this might happen. For example, a site may have content that may be viewed at the same URL in multiple languages depending on the browser settings. Another example would be content that gets updated every few minutes or so.

Whatever the means, search engines typically index only one copy of a specific URL once every few weeks or so. Therefore, if a search engine indexes your English content at a given URL, the same search engine will not index your Spanish content at the same URL (during the same indexing period). And if your content is frequently updated, the search engine's copy of your content will not be fresh. A search engine prefers that the visitors to a particular URL see the same content its spider saw.

Jill: Sounds like sites with dynamic content have an uphill climb when it comes to the search engines. So what can we do to help them get indexed?

Alan: The general answer is to give each search engine what it wants: unique core content at a unique URL, plus the same core content seen by all visitors.

But I'm guessing you want specifics. So here they are!

1. Use static URLs to reference dynamic content.

If a search engine sees a static URL, it is more likely to index the content at that URL than if it found the same content under a dynamic URL. Therefore, you can turn your dynamic URLs into static URLs despite the fact that you are serving dynamic content. There are a number of ways of achieving this, and your method will vary depending upon your server and other factors. To go into all of these methods is beyond the scope of this interview; however, you can visit the following sites for two popular servers:

Apache
ASP

2. Link to dynamic URLs from static URL content.

With limited resources, it may prove difficult or impossible for you to implement a solution based on static URLs. Don't worry! There are other things you can do.

Over the years, the engines have tried to find ways of crawling dynamic content while avoiding dynamic spider traps. One technique they use is crawling dynamic URLs that are linked to from pages with static URLs. For example, if you give your site map page a static URL, but have links to dynamic URLs within its content, there's a good chance that the leading engines will crawl those links. If they like the content they find there, they will index that content. The search engines' reasoning here seems to be, "If you're prepared to link to this content, then so are we."

You can reinforce this reasoning by negotiating links to your dynamic URLs from pages on other sites (especially high-quality pages which are already indexed). Again, the search engines' reasoning here is "If other sites are prepared to link to your site, then so will we." If others won't link to your dynamic content, that might give you some idea why search engines won't either! If it proves impossible to get links to your dynamic content from other sites, then you can't expect a search engine to link to your site either.

3. Pay for inclusion whenever possible.

AltaVista, Ask Jeeves/TEOMA, FAST and Inktomi offer one or more means of paying for individual URLs to be spidered. You can use these paid-inclusion programs to get your dynamic URLs indexed. Paid-inclusion programs only affect inclusion and do not influence ranking, so it is still important to make sure your dynamic content is well optimized. For more details see the Add-URL pages of the respective search engines.

Conclusions:

1. Search engines have problems creating links to dynamic content.

2. If you can recognize these problems, you are halfway to getting your dynamic content indexed.

3. Where practical, use static URLs to reference dynamic content.

4. Otherwise, try to ensure your dynamic URL is linked to by content referenced by static URLs.

5. Consider using paid-inclusion programs.

Jill: Thanks for your answers and your time, Alan!


August 11, 2003





CEO and founder of High Rankings®, Jill Whalen has been performing search engine optimization since 1995 and is the host of the free High Rankings Advisor search engine marketing newsletter, author of "The Nitty-gritty of Writing for the Search Engines" and founder/administrator of the popular High Rankings Search Engine Optimization Forum. In 2006, Jill co-founded SEMNE, a local search engine marketing networking organization for people and companies in New England.

High Rankings is an internationally recognized search engine optimization firm located in Framingham, MA specializing in search engine optimization, SEO consultations, in-house training, site audit reports, search marketing seminars and workshops. High Rankings has a 100% success rate for substantially improving client rankings and targeted traffic.

Jill speaks at national and international conferences and has been writing about SEO and search marketing since 2000. She's been quoted in such publications as The Wall Street Journal, U.S. News & World Report and The Washington Post. Her articles have appeared in numerous print magazines and online websites including CIO Magazine, CMS Focus, The Internet Marketing Report, ClickZ, WorkZ, Inc.com, Entrepreneur, Lycos Small Business, WebProNews, SiteProNews and others. Jill has also appeared on many online and offline radio programs such as Entrepreneur Magazine's E-Biz Radio Show, SearchEngineRadio and the eMarketing Talkshow.





Search Engine Guide > Jill Whalen > Optimizing Dynamic Content for Search Engines