I find from my server logs that there are some requests for a robots.txt file from some search engines. However, I don't have a robots.txt file. Do I need to have such a file? Will my ranking be affected if I don't have such a file?

Evan Kendall


Hi Evan,

A robots.txt file is used to prevent somes pages in your site from being indexed by the search engines. When a search engine spider visits a site, it first looks for a robots.txt file. If it doesn't find such a file, it will spider all the pages in your site. If it does find such a file, it will read this to find out which pages in your site you do not want it to spider. Then, it will spider only those pages which you have not specifically disallowed.

Now, normally, you want the search engines to be able to spider all the pages in your site. Hence, you generally don't require a robots.txt file. The fact that the search engines are looking for such a file and it is not present in your site will not affect your rankings in any way.

Here are a couple of situations when you may need a robots.txt file:

i) You will need a robots.txt file if you have created many similar looking doorway pages, each page meant for a different engine. Since the search engines frown upon the presence of too many similar pages in a site, you need to create a robots.txt file to prevent the search engines from being able to spider the pages which are not meant for it. However, I don't recommend that you create such similar looking doorway pages in the first place.

ii) You will need a robots.txt file if you have used Meta Refresh tags in some pages. While there are plenty of legitimate reasons why you may need to use Meta Refresh tags in certain pages, the search engines generally frown upon sites using Meta Refresh tags because people often use it for spamming. Hence, you need to prevent the search engines from spidering the pages containing Meta Refresh tags. You can do this by using a robots.txt file.

For more information on how you can create a robots.txt file, see my article covering this topic.

There is a software called Robogen which helps you automate the creation of the robots.txt file. You can read about this software at my site.


I recently submitted my site to Yahoo! using its Business Express Program, and was accepted. However, I found that instead of placing my site in one of the main Yahoo! categories, it placed me in a Regional category (the region in which my site is located). What can I do to get listed in one of the main Yahoo! categories?

Janet Kingston


Hi Janet,

Firstly, you need to ask yourself a question - is your product or service intended primarily for the region in which your site is based, or will it be useful for people living in other regions as well? If the answer is that what you are providing is primarily useful for people living in your region, then there's not much you can do about getting listed in one of the main Yahoo! categories - Yahoo! will probably never agree to list your site in one of the main categories.

However, if people living in other regions can also use your product or service, then you need to send a very polite appeal to Yahoo! within 30 days from the time that Yahoo! sent you the acceptance notice. In your appeal, thank them for listing the site (even if you don't feel all that grateful to them :-)). Then state that you have noticed that your site has been placed in a regional category. Then tell them that your products or services are not restricted to serving only the people in your region, but that they can be used in other regions as well. Give them concrete examples regarding this. Tell Yahoo! the exact regions where your existing customers are located. If you have had customers from other countries, tell Yahoo! the names of those countries. In short, do everything possible to ensure that Yahoo! is convinced that your site is not regional in nature. After this, end your appeal by requesting them to evaluate your site again, and to add it to the main categories to which you had submitted the site because doing so will improve the accuracy of the listing of your site in Yahoo!.

We have had a few clients who had initially been listed in one of the regional categories even though we had submitted the site to the main categories. However, when we appealed to Yahoo! using the format outlined in the earlier paragraph, they were all accepted in a main category in Yahoo!.

Another thing that you should keep in mind when you appeal your listing is that Yahoo! is not concerned about how much traffic your site's listing will bring you - they are only concerned about building up an accurate directory of web sites. Hence, whenever you appeal your listing in Yahoo!, don't tell them that your site's current listing won't bring you much traffic (even though that is the reason you are appealing). Giving such a reason will all but ensure that your appeal is rejected. Instead, you must tell Yahoo! why the current listing is inaccurate and why what you suggest will make the listing more accurate.


I have the #1 ranking in the Open Directory for my most important keyword. However, my traffic analysis program shows that I haven't received a single hit from the Open Directory for this keyword. Can you tell me why?

Kim Nicholson


Hi Kim,

The Open Directory ( itself doesn't have too many people who use it as their default source for finding information, i.e. the Open Directory is not a very popular site in its own right. Thus, it is not surprising that your site, even though it is ranked #1, hasn't got any traffic from the Open Directory. In fact, whatever little traffic that you get directly from the Open Directory will generally be from people browsing around the categories (rather than using the search feature). Even in this case, a large proportion of this traffic will be editors of the Open Directory who are checking to see whether your site is still working and whether your site's listing in the Open Directory matches the actual content of your site.

Hence, you should not be too concerned about the fact that you are not getting traffic from the Open Directory. Instead, the importance of the Open Directory lies in the fact that it provides the results for search engines like AOL, Lycos etc. Google also gives a lot of emphasis on sites being listed in the Open Directory. Also, the Open Directory acts as the default directory provider for these and many other engines and sites. Thus, while the Open Directory is not important as a source of traffic in its own right, it is important because an optimum listing in it leads to a lot of traffic from other sites.

Incidentally, the different engines which use the Open Directory to drive their search results all have their own algorithms which they use to rank sites from the Open Directory, which indicates that a site which is ranked well in the Open Directory for a particular keyword may not be ranked well in AOL and Lycos for the same keyword.
September 14, 2001

Sumantra Roy, president of 1st Search Engine, is one of the most respected and recognized search engine positioning specialists on the Internet. Sumantra's articles on search engine optimization have been widely published and quoted in different web sites and newsletters on the Internet. His unique formula for establishing the most appropriate keywords for a site has been used by WordTracker, the most popular keyword research tool on the Internet. This formula is also used by many other search engine optimization companies for establishing the most appropriate keywords for their clients' sites.

Search Engine Guide > Sumantra Roy > Robots.txt File, Changing Categories in Yahoo!, ODP Traffic