The field of search engine optimization is a constantly evolving industry with important changes being made almost daily.
In the beginning, search engine optimizers focused on one thing only: rankings. Tracking of actual sales or conversion rates was almost unheard of.
Slowly, search engine marketers began to realize that all of the 1s in the world wont help if they dont convert to traffic, and all the traffic in the world wont help if it doesnt convert to sales.
So, search engine experts began analyzing their log files and tracking sales. With todays technology, Web site owners can now tell which of their marketing campaigns are truly effective and which need to be replaced by a different campaign. In other words, through log analysis, you can analyze your human traffic and their visits to your Web site.
But what about your spider traffic?
One part of log analysis that has remained surprisingly lacking, both in terms of content and accuracy, is spider analysis.
Though traffic analysis programs may look at spider activity, the information often isnt detailed enough or presented in a format to do you much good. Also, spider and robot analysis is acknowledged as being a main culprit for inaccurate log analysis measurements.
Therefore, the need for detailed spider analysis has begun entering the minds of search engine marketers.
So, what is spider analysis?
You know that when you submit a Web page to an engine for indexing, the engine sends a spider to your site to index the contents of the page. Spider analysis is simply analyzing the search engine spider visits to your site.
Through effective spider analysis, you can learn the following information about your site in a concise, easy-to-read format.
Another important issue in spider analysis is robots.txt files.
Though the object of search engine marketing is to help spiders find all the pages on your Web site, there will be times when you want to keep spiders out of certain pages. You can do this with a robots.txt file.
What is a robots.txt file?
A robots.txt file is a text file that is placed on your server that instructs the search engine spiders not to crawl or index certain sections or pages of your site.
But why would you want to keep the spiders out?
Lets say that youre creating a new page for your site, and youve placed the page online while you tweak and edit it. The page isnt ready for visitors, so you wouldnt want it indexed yet. Or, lets say youve placed some employee guidelines on your site. The guidelines are of interest only to your employees, and theres no reason for the public to view them.
Using a robots.txt file, you can keep the spiders out of those pages.
Therefore, when working with spiders or robots, you want to be able to:
How do you create a robots.txt file?
Creating a robots.txt file manually is tricky at best. One little mistake will make the entire file invalid, and youre opening your Web site up to spiders operating on their own free will. Plus, if youve created engine-specific pages with similar content, the spiders may discover those nearly duplicate pages, and you could find yourself in trouble for spamming.
While there have been software programs on the market to create robots.txt files, thats been their sole function. Spider analysis hasnt been a part of their features, until now.
Robot-Manager Professional Edition is a software program that concentrates totally on spider visits to your site.
Recently, I had the good fortune of reviewing a new software program that performs two major functions: spider analysis and robots.txt creation.
In a nutshell, Im amazed, and I have to predict that the software, Robot-Manager Professional Edition, will rapidly become a must have tool for search engine marketers.
To begin with, the program is easy to use. I dont know about you, but I dont have time to learn a complicated software program. Though the program is intuitive and doesnt require the reading of a complicated users guide, youll still find detailed help topics that can answer any questions.
The program begins with Step 1, where you choose which spiders you want to work with. In Step 2, you download your file directory tree, which is where you begin to tell the program which files you want to keep spiders out of. The robots.txt file is automatically created in Step 3, and you can instantly upload the file to your directory. The program allows you to edit the file manually as well.
In Step 4, the program gets down to the real meat of the analysis: analyzing your log files for spider visits. When downloading and analyzing the log files from the domain I worked with, it automatically added 30 new spiders to the spider list.
On the Spider Visits page, I found it particularly helpful to view the visits by Web page. In this manner, I could quickly see which spiders have visited which pages of the site and when. Just think of the value of this information when working with clients. You can prove to them that spiders have visited their sites, even if the pages arent yet indexed. You can also view spider visits by date or by spider.
Also, international spiders are included in the program, which is ideal for those sites that are aiming for a corner of the international market.
Where can you go to see this program for yourself?
Visit Website Management Tools and look for Robot-Manager" 3.0 Professional Edition. Though the site also offers a Standard Edition of the software, I strongly advise you to look at the Professional Edition instead. Why? Because the Standard Edition doesnt include the spider analysis portion of the software, which is a must have, in my opinion.
You can download a trial version of the software at no cost, then test drive it before purchasing. With the trial version, only 5 spiders can be selected and 20 spider visits picked up from the log file.
In Conclusion . . .
As the search engine marketing industry continues to move forward, three major categories of tools or services need to be considered:
If your current search engine marketing plan doesnt cover each of those crucial areas, you need to look into expanding your tool arsenal. The information you can gain by analyzing your human and spider traffic will prove invaluable to you as you work toward strengthening your online business.
Robin Nobles is the Co-Director of Training of Search Engine Workshops, where they teach "hands on" search engine marketing workshops in locations across the globe. They also provide a networking community for SEOs called The World Resource Center for Search Engine Marketers and have expanded their workshops to Europe with Search Engine Workshops UK.
Copyright © 1998 - 2020 Search Engine Guide All Rights Reserved. Privacy