Webmaster World Removes Themselves From the Search Engines

In an industry obsessed with search engine optimization and marketing, it seems like a crazy idea. Purposely block all spiders from visiting your site and allow your listings to drop out of engines like Google, Yahoo! and MSN. But that's just what is being done by Webmaster World owner Brett Tabke.

The problem is that Webmaster World has become so large and so popular that it's suffering from an overload of rogue spiders that are taking a serious toll on the server. Says Brett: "We have pushed the limits of page delivery, banning, ip based, agent based, and down right cloaking to avoid the rogue bots - but it is becoming an increasingly difficult problem to control." So, in a bold move that has sparked quite a bit of criticism and commentary, Brett had decided to ban all spiders from crawling the site.

Barry Schwartz has the full story in an interview with Brett at SEO Roundtable.

From the interview:

One thing that sets Webmaster World apart from all other similar sites, is the ease with which we can be crawled. There are no CGI parameters on url strings and all off-the-shelf bots can index the site. I can write a 15 line perl program in 5 minutes that will download the entire site - even with cookie support. That same thing can not be said about sites that are not freely crawlable (like other forums and auction sites with cgi based or non standard urls).

The change was for us to require cookie support via member login. That action mandates either allowing the approved big search engine crawlers to feast on a login page instead of page viewing several million pages before they realized the site was 100% different than before. The easiest solution to that is to set a robots.txt ban on all crawlers.

I knew it would be a controversial action. In such cases, it is always better to bring up the subject yourself or least people get the wrong impression that it was by no action of your own. I just threw up the post as a marker so that people knew we'd taken the action ourselves and I would come back later with more information after things settled down a bit. We had started down this road about mid-july when we began blocking many of the major crawlers.

> Why was this the last course of action?

We've tried every thing to stop the bots. Once we got up to several thousand ip's in the system ban list, it was having a serious effect on system performance. We also were occasionally into a situation where we would ban an IP and then that IP would get recycled to another member that had nothing to do with a download attack. It is hard to block an IP such as an AOL IP, because you block several million users using that IP via the AOL proxy cache.

Danny Sullivan has extensive commentary on the issue at Search Engine Watch and the Search Engine Watch forums also have a discussion thread. Discussion is also (obviously) taking place at Webmaster World.