Have you ever wondered how far down into your site the search engine spiders are going? Are the pay inclusion spiders coming at regular intervals like theyre supposed to?
Wouldnt it be great to see which spiders are visiting your site and which pages theyre crawling . . . at a glance?
Robot Manager Pro is undoubtedly one of the most helpful software programs Ive seen in a long time. It actually serves two purposes:
What is a Robots.txt File?
A robots.txt file is simply a text file that disallows (or keeps spiders out) of certain pages or areas of your Web site. For example, if you have a section of your site thats devoted to your personnel, you can keep the spiders from finding and crawling that entire section by including it in a robots.txt file.
If youve ever tried to create a robots.txt file by hand, you know that its nothing short of tedious, and its so easy to make a mistake. So having a program create the file for you automatically is ideal.
However, whats really exciting to me about Robot Manager Pro is how it monitors spider traffic. But first, lets look at the software as a whole.
At Look at Robot Manager Pro
The program is simple to use. On the left-hand side of the screen, youll see numbered areas. On the right-hand side of the screen, youll see a Quick Help section that explains how to use that portion of the program.
What youre doing in steps #1-3 is creating your robots.txt file. Youll tell the program which areas or pages that you want to keep the spiders from crawling, and the software will create the robots.txt file for you and upload it to the root directory of your Web site.
Keep in mind that you dont have to create a robots.txt file if you dont want or need to. You can simply use the spider monitoring portion by itself.
Now Lets Get to the Fun Stuff
As I stated previously, the most exciting part of the program to me is the way it monitors spider traffic.
The first time you download your log files to analyze your traffic, it wont be quite as interesting as it gets each additional time you use the program.
Every time you use the program, youll open the initial file you created, hit refresh to download the latest log files, and let the program update your spider visits.
This screen shot shows a listing of the spiders that have visited the example site.
If we were to click on one of the spiders, such as Googlebot, wed see this screen:
Notice how many pages from this site that Google has spidered, and this screen shot certainly doesnt show them all. Google is a deep crawling engine, and it really can find most of your Web pages if you provide text links to them.
If we were to click on any of the actual Web pages, wed see a screen that tells us when the spider has visited the page.
In this example, look at the number of times that Googlebot spidered this particular page over the last few months.
Can you see the value in having this type of information?
· If an engine hasnt found one of your important pages, maybe you need to add more links to that page from pages that are getting spidered more frequently.
· When you create new pages, youll be able to see at a glance when the spiders find and crawl those pages.
· If you use pay inclusion, youll be able to monitor whether the spiders are visiting like theyre supposed to.
· Youll know exactly which pages of your Web site are being spidered by which search engines, and when.
In Conclusion
If youre interested in tracking spider visits to your site, Robot Manager Pro is a must have software program for you. Its fast and easy to use, and it provides key information about how the spiders are crawling your Web site.
Visit the Robot Manager Pro Web site at http://www.websitemanagementtools.com/robot.html to learn more or to download a free trial version. They also offer a regular version of Robot Manager, but it doesnt monitor spider traffic. So, be sure to click on the Professional Edition on the left tool bar.
|
Robin Nobles is the Co-Director of Training of Search Engine Workshops, where they teach "hands on" search engine marketing workshops in locations across the globe. They also provide a networking community for SEOs called The World Resource Center for Search Engine Marketers and have expanded their workshops to Europe with Search Engine Workshops UK.
Search marketing information for small business owners.
Fetching the best small business news.
A friendly place to share small business ideas and knowledge.
Small business support through education, resources and community
The directory of the best small business sites and tools.
Copyright © 1998 - 2020 Search Engine Guide All Rights Reserved. Privacy
FreeFind Site Search Engine - FreeFind adds a "search this site" feature to your website, making your site easier to use. FreeFind also gives you reports showing what your visitors are searching for, enabling you to improve your site. FreeFind's advanced site search engine and automatic site map technology can be added to your website for free.
Buy UPC Codes
Get your products listed online!