Advisor subscriber Lia Gonzalez had a number of quick questions for me, so instead of posting her entire email and repeating the questions twice, I've simply posted them with my answers underneath each one.

Q. Do search engines crawl other search engines' databases to "steal" their directories in order to enhance their own?

A. Search engines do crawl through directories such as Yahoo, DMOZ (The Open Directory), GoGuides, Gimpsy and JoeAnt and often add those results to their own. Any search engine or directory that does not want their results to be crawled should place a robots exclusion in their robots.txt file on their server. I believe that most of the major search engines have these in place. (See Google's here: http://www.google.com/robots.txt)

 

Q. Will being well placed in a top engine like Google help the site eventually make it into the others?

A. Being placed in directories such as the ones mentioned above will eventually get you listed in the other spidering search engines such as Inktomi, AltaVista, Lycos etc. However, being listed in other search engines probably won't help. That said, many search engines share the same database. For instance, once you're listed in Google, those results are picked up by AOL and Yahoo, because they have a deal with Google to use their listings. The FAST database is used by Lycos and many international search engines, as well as their own AllTheWeb.com site. The Inktomi database is used by MSN, and is also "backfill" for numerous directories and search engines.

 

Q. Is there a "back door" to appear in DMOZ? It was easy to submit in Google and impossible to submit in DMOZ; its interface just stagnates. I have tried to contact their Webmaster but never got an acknowledgment from him.

A. No, there's no back door into DMOZ. Unless you can find an unscrupulous editor, of course! Hopefully, most of them have been removed over the years. By the way, DMOZ doesn't have a "Webmaster" to contact about your listing. DMOZ is comprised of thousands of volunteer editors. If you have a problem getting listed, you should contact the editor of the category where you submitted your site. However, please note that it can take many, many months to get listed in DMOZ, and many times you will never be listed. Be sure to read their guidelines very carefully, and create your titles and descriptions just as carefully. DMOZ is inundated with "spam" submissions and it takes them a long time to separate the good from the bad. Personally, I don't worry about DMOZ too much any more. I believe they are on their way out. The site is down most of the time when I try to visit it, and it seems that perhaps AOL has given up on them completely.

Sure, a listing with DMOZ is still a great one to have, but don't obsess over it if you can't seem to get listed. Submit to one of the many other fine directories instead. Try JoeAnt, GoGuides and Gimpsy for starters. Make sure you follow their rules and don't spam them or your submissions will be ignored or rejected. (Please note that some of these directories charge a fee for expedited review, and/or require you to become an editor before you can submit your site. Oh, and watch out for the GoGuides' new colors; they nearly blinded me when I visited the other day. Yuck!)

 

Q. Which engines are the best to submit to?

A. You don't have to submit to any spidering search engines. They will crawl the Internet and find your site as long as it is listed on other sites such as the directories mentioned above.

 

Q. Won't the engines that ONLY add paid submissions to their databases lose credibility from users? Users will get results not from the best sites, but from the richest that can afford to be included. As a user I would not consider using any search engine that has a paid-submission database.

A. There are no major search engines that I know of which *only* accept paid submissions. Plus, it's not necessary to pay for inclusion to any of the current crop of search engines, as long as you don't mind waiting to be added the free way. If you're in a hurry, then pay. If you want to tweak your pages every 48 hours, then pay. If you don't mind waiting 6 - 8 weeks to be spidered and you won't be making changes that often, then wait for them to find your site. The spiders are doing a very good job of crawling these days, which is as it should be!

I agree with you that search engines would lose credibility (and relevance) if they only allowed paid inclusions. This is why all of the major search engines still add pages for free. They understand that having a database of only paid sites won't be very good for their users. Certainly, the search engines have to make money too, however, and it's nice that many of them offer a paid-inclusion option for those who don't want to leave things to chance.

Hope this helps!

Jill