The Importance of Spider Analysis in a Well-Rounded Search Engine Marketing Plan

The field of search engine optimization is a constantly evolving industry with important changes being made almost daily.

In the beginning, search engine optimizers focused on one thing only: rankings. Tracking of actual sales or conversion rates was almost unheard of.

Slowly, search engine marketers began to realize that all of the 1’s in the world won’t help if they don’t convert to traffic, and all the traffic in the world won’t help if it doesn’t convert to sales.

So, search engine experts began analyzing their log files and tracking sales. With today’s technology, Web site owners can now tell which of their marketing campaigns are truly effective and which need to be replaced by a different campaign. In other words, through log analysis, you can analyze your human traffic and their visits to your Web site.

But what about your “spider” traffic?

One part of log analysis that has remained surprisingly lacking, both in terms of content and accuracy, is spider analysis.

Though traffic analysis programs may look at spider activity, the information often isn’t detailed enough or presented in a format to do you much good. Also, spider and robot analysis is acknowledged as being a main culprit for inaccurate log analysis measurements.

Therefore, the need for detailed spider analysis has begun entering the minds of search engine marketers.

So, what is “spider analysis”?

You know that when you submit a Web page to an engine for indexing, the engine sends a spider to your site to index the contents of the page. “Spider analysis” is simply analyzing the search engine spider visits to your site.

Through effective spider analysis, you can learn the following information about your site in a concise, easy-to-read format.

• Has your site been spidered?
• If so, by which engines?
• When did the spiders visit?
• Which directories and pages did they visit?
• Are certain pages getting respidered more often, signaling their importance to the search engines?
• Are certain pages not getting spidered at all?
• Are the spiders indexing inappropriate content?
• Are the spiders getting everything they want and need, or are they receiving error messages?
• Was your site spidered within the specified time agreed upon in the pay inclusion programs you’re participating in?
• Is your site getting respidered on a regular basis, as agreed upon in your participating pay inclusion programs?

Another important issue in spider analysis is robots.txt files.

Though the object of search engine marketing is to help spiders find all the pages on your Web site, there will be times when you want to keep spiders out of certain pages. You can do this with a robots.txt file.

What is a robots.txt file?

A robots.txt file is a text file that is placed on your server that instructs the search engine spiders not to crawl or index certain sections or pages of your site.

But why would you want to keep the spiders out?

Let’s say that you’re creating a new page for your site, and you’ve placed the page online while you tweak and edit it. The page isn’t ready for visitors, so you wouldn’t want it indexed yet. Or, let’s say you’ve placed some employee guidelines on your site. The guidelines are of interest only to your employees, and there’s no reason for the public to view them.

Using a robots.txt file, you can keep the spiders out of those pages.

Therefore, when working with spiders or robots, you want to be able to:

• Create a robots.txt file quickly and easily;
• Use a robots.txt file to present optimized pages to specific engines. For example, using a robots.txt file, you can focus English language robots onto the relevant pages and direct robots from international search engines to the localized content areas of your site;
• Send e-mail harvesting programs away from your site to keep your e-mail spam down;
• View highlighted pages requested in error by the spiders; and,
• Direct search engine spiders to relevant areas of your Web site.

How do you create a robots.txt file?

Creating a robots.txt file manually is tricky at best. One little mistake will make the entire file invalid, and you’re opening your Web site up to spiders operating on their own free will. Plus, if you’ve created engine-specific pages with similar content, the spiders may discover those nearly duplicate pages, and you could find yourself in trouble for spamming.

While there have been software programs on the market to create robots.txt files, that’s been their sole function. Spider analysis hasn’t been a part of their features, until now.

Robot-Manager Professional Edition is a software program that concentrates totally on spider visits to your site.

Recently, I had the good fortune of reviewing a new software program that performs two major functions: spider analysis and robots.txt creation.

In a nutshell, I’m amazed, and I have to predict that the software, Robot-Manager Professional Edition, will rapidly become a “must have” tool for search engine marketers.

To begin with, the program is easy to use. I don’t know about you, but I don’t have time to learn a complicated software program. Though the program is intuitive and doesn’t require the reading of a complicated user’s guide, you’ll still find detailed help topics that can answer any questions.

The program begins with Step 1, where you choose which spiders you want to work with. In Step 2, you download your file directory tree, which is where you begin to tell the program which files you want to keep spiders out of. The robots.txt file is automatically created in Step 3, and you can instantly upload the file to your directory. The program allows you to edit the file manually as well.

In Step 4, the program gets down to the real “meat” of the analysis: analyzing your log files for spider visits. When downloading and analyzing the log files from the domain I worked with, it automatically added 30 new spiders to the spider list.

On the Spider Visits page, I found it particularly helpful to view the visits by Web page. In this manner, I could quickly see which spiders have visited which pages of the site and when. Just think of the value of this information when working with clients. You can prove to them that spiders have visited their sites, even if the pages aren’t yet indexed. You can also view spider visits by date or by spider.

Also, international spiders are included in the program, which is ideal for those sites that are aiming for a corner of the international market.

Where can you go to see this program for yourself?

Visit Website Management Tools and look for Robot-Manager" 3.0 Professional Edition. Though the site also offers a Standard Edition of the software, I strongly advise you to look at the Professional Edition instead. Why? Because the Standard Edition doesn’t include the spider analysis portion of the software, which is a “must have,” in my opinion.

You can download a trial version of the software at no cost, then test drive it before purchasing. With the trial version, only 5 spiders can be selected and 20 spider visits picked up from the log file.

In Conclusion . . .

As the search engine marketing industry continues to move forward, three major categories of tools or services need to be considered:

• Web page checking, submission, and ranking tools;
• Web traffic analysis tools for analyzing human visitors to your site; and,
• Spider analysis tools for analyzing spider activity.

If your current search engine marketing plan doesn’t cover each of those crucial areas, you need to look into expanding your tool arsenal. The information you can gain by analyzing your human and spider traffic will prove invaluable to you as you work toward strengthening your online business.

February 28, 2002