Did Search Engines Get a Bum Rap?

It's no wonder SEO marketing strategies have gained momentum as marketers realize the power of search engine positioning. It's been documented since 1994 that search engine traffic is relatively cost-effective and gets good results. Of course, search engines are no panacea, and should be used as part of a balanced marketing strategy. But the fact remains that in some cases, search traffic is more effective than banners, email, and other Web marketing techniques.

That's because search-driven visitors are self-selected, thus pre-qualified. It's been said that search traffic is the most qualified and potentially convertible traffic you can find. It might have to do with timing, as users are actively looking for your products and services in search databases before clicking to your site in buying mode.

Commercialization of the Web

Search engines have struggled over the past year or so, with high overhead and dwindling portal revenues. Some have almost gone out of business, like Go and Excite, offering fewer services, closing international sites, and depending on content from partners. While search engines started off as objective databases, the erosion of ad revenues has resulted in more commercialization.

This week, search engines have been dealt a tough blow, perhaps one that might have been avoided. Commercial Alert, a consumer watchdog organization started by Gary Ruskin and Ralph Nader, filed a deceptive advertising complaint with the Federal Trade Commission against eight search engines. The complaint alleges that placing ads in search engine results without clear disclosure is unlawful, claiming that such listings "look like information from an objective database selected by objective algorithm� but are paid ads in disguise."

The search engines and directories cited in the complaint are iWon.com, Lycos, MSN.com, Netscape, AltaVista, Direct Hit, HotBot and LookSmart. GoTo.com was not named because it clearly identifies its paid listings and even posts the amount paid. Notably, Yahoo! and Google were also not named.

The End of Free

The problem is that the search industry, like all other nascent Web marketing businesses, is in the early development stages of the marketing cycle. There have been many changes so far, and there will be more ahead before the industry matures. Its first business model, ad-funded publishing, has failed. So in order to remain afloat and still provide free services to users, the search engines have been trying out all kinds of new revenue schemes.

The first to start charging for inclusion were directories such as Yahoo! and LookSmart, which offered a fee option for faster editorial review (e.g., Direct Express), although listing is not guaranteed. Then came fee options for "deep listings" (e.g., LookSmart Subsite), which doesn't guarantee positioning but can increase the probability of appearing in response to a wide range of queries.

But potential trouble began brewing last year, as most databases started including GoTo paid placement results in their main search results, using ambiguous labels such as "featured listings," "featured sites," "partner search results" -- everybody had a different label. Sometimes paid listings were shown at the top, other times below editorial content. With benefit of hindsight, we can see there was no consistency, either in the way these links were identified by the different engines, or where they were placed in the results. Worst of all, there was no clear explanation that these were paid links.

There were rumblings in the press. Back in November, Elizabeth Weise of USA Today wrote "Web Sites Pay to Propel Search Engines; Fees for Detailed Listings Raise Doubts About Where the Links Are Taking You." I raised the controversy on ClickZ with "The Homogenization of Web Search" in November and "Paid Search Links Push for a Piece of the Pie" in December. More recently, search guru Danny Sullivan of Search Engine Watch wrote "Buying Your Way in to Search Engines" in May. Saul Hansell of The New York Times wrote "Clicks for Sale; Paid Placement Is Catching On in Web Searches" June 4. Verne Kopytoff of the San Francisco Chronicle wrote "Searching for Profits; Amid Tech Slump, More Portals Sell Search Engine Results to the Highest Bidder" June 18.

So while controversy brewed, it was largely ignored. I suppose if the industry had its act together with a responsible trade organization, this may never have happened. Industry standards would have forced disclosure and conformity. But the industry has been slow to organize. Just look how long it took the IAB to spell out standard banner sizes.

Is There a Solution?

Many industry players defend the practice of allowing paid listings to rank in public search engines, saying there should be some middle ground the search industry can move toward, to encourage Web sites to pay for listings without affecting their rankings in search results. As web sites proliferate exponentially, search engine databases get increasingly harder too handle and more expensive to maintain. Public search engines can't be free any more, someone has to pay the bills.

Here's how paid inclusion programs can help offset the cost of more frequent spidering, which benefits the engines, marketers, and users alike. Inktomi has a program called Index Connect, where small companies can pay as little as $30 for Web listings. Such payments allow search engines to extend their reach and update their databanks more frequently without having to rely on banner ads. This can be done without altering search results.

And here's how paid placement can be made legitimate: by clearly labeling it for what it is and standardizing the labels. The search engines might also collaborate to get a third-party survey of users, learning how they feel about ranking paid links on public search engines. Something needs to be done to clear up this controversy before it festers.

With the industry struggling to survive, it's disappointing to see companies take a hit like this when they're trying to sustain a business model. On the other hand, it's important that the integrity of search results be maintained and that paid links be clearly denoted. It's a no-brainer that search companies need to make a profit to offer basic Web services such as public search.

Let's hope the FTC handles this with finesse and gives these companies a chance to clean up their act without causing too much pain. Paid placement is here to stay, for better or for worse. With content sites moving to subscription models, the days of free are over, and Web information is a commodity to be paid for. I'm sure consumers would rather see it paid for this way, than to see search services forced to charge users for information.
July 23, 2001