Search Engine Guide
Search The Internet: 

Traffick Logo Article provided with permission by
Traffick - The Guide to Portals.
© 2000

Searchonomics: A New Era in Human-Guided Web Search?
By Andrew Goodman (from Traffick Monthly newsletter) - August 29, 2000

Legend has it that a major plank in Ronald Reagan's economic platform - the Laffer Curve - was sketched out for him on a napkin. On this foundation, Reagonomics became the ovverriding theme of a decade of US fiscal policy. A vast array of reforms flowed from some basic reasoning that essentially argued that if you put a luxury tax on boats, sales will decline, and the luxury boat industry will dry up.

A lot of powerful ideas would appear to originate this simply. 1998 was a year in which everyone in the Internet business decided that there was room to improve on Yahoo! - in particular, its directory, which, critics charged, couldn't scale with the growth of Internet content. The overworked team of Yahoo! editors let "link rot" set in, and ignored many submissions.

Can't anyone be a "surfer"?

It's telling that Yahoo!'s job descriptions refer to directory editors as "surfers." If you're thinking "just about anyone could do that," you're not far from wrong. And in fact, there is a huge thirst amongst heavy Internet users to get more involved in rating sites, offering advice, publishing articles, and ranking products. A host of innovative business models are appearing on the scene today, promising to give a platform, and sometimes some pecuniary rewards, to these legions of enthusiasts and experts. This new type of knowledge worker may potentially fuel the next leap forward in information delivery on the Internet.

One team of inventors, led by Steve Thomas, a software developer and aficionado of cognitive science, sketched out some principles for the ultimate web search resource "in a one-page concept paper" in February of 1998. Barely larger than the proverbial napkin, this paper kicked off a process of planning and development for a new system of "Searchonomics" under the company name Wherewithal.

Founding principles

Thomas had been Netscape's Product Manager, Platform - responsible for Java/Javascript, DHTML, and inventing something called "layers." With Netscape since shortly after its IPO in 1995, he left in 1998 to start Wherewithal. Along with co-founder and CIO Darren Skinner, Thomas sketched out key principles for a new web search resource to end all resources:

  • Scalability. Observing Yahoo! and other human-managed directories, Thomas and his team believed that most of them were designed with assumptions of the "small scale Internet" in mind. After all, Yahoo! began as Jerry Yang's Guide to WWW, a helpful guide to Internet content in different subject areas. It wasn't all that hard to build. Set up an arbitrary taxonomy, get a few people working on it, and reasonable coverage of the web was possible. As the Internet expanded, this model, in Wherewithal's view, became obsolete.

  • Human element. One thing Yahoo! did have going for it was the judgment employed by category editors. Robots can be inflexible, and do a poor job of acting as "gatekeepers" and judges of what's good and what isn't. The raw knowledge available on the Internet needs to be digested by "informediaries" if it is to be useful.

  • The possibility of multiple taxonomies. If you think about a Yahoo! category, or for that matter, the category structure in any human-edited directory or web guide (eg. Looksmart, Open Directory, or About), the choice of how to organize the resources in a given field - which to highlight, which to downplay, which sites to accept, and which to reject - is made by a single editor. Even if there is a committee system in place in an editorial organization, the end user only sees one category structure and one opinion as to what resources about sports cars (for example) are most useful. Such an arbitrary method for a medium which is supposed to be about flexibility, self-publishing, even chaos! The Wherewithal plan needed to come up with a way for the end user to substitute their favorite editor (a so-called Second Opinion) in place of one which they felt wasn't suitable for their needs. This builds accountability into the system. There is no such thing as a "bad" editor (the users will determine that), and therefore command-and-control administration of the editorial personnel isn't needed.

  • A sustainable incentive system. Why would anyone be a category owner in the first place? The work of infomediaries has value. Over time, some analysts predict, the compensation of such infomediaries will adjust to closer reflect the value of that work - in part, through choices made by the infomediaries themselves. We're well past the days when we simply volunteered advice in Usenet newsgroups without any expectation of reward. The act of volunteering advice or pointing people towards useful resources isn't going to go away, but most infomediaries and consultants today have a sharper sense of how soon to put their "clients" on the "meter." In a manner similar to About, which pays its guides a percentage of the revenues generated from their Guide Sites, Wherewithal's Searchonomics concept pays editors a portion of the revenues from targeted advertising in their directory categories. Editors who attract more users through the high quality of their work, or simply through the act of editing in more marketable categories, stand to make more income.
Newhoo Built a Valuable Asset Quickly

As Thomas tells it, Wherewithal became aware of the advent of Newhoo (now Open Directory) about six months into the life of their own project. Wherewithal was diligently working on the back end - the technical and operational system which would give the project legs. By contrast, Newhoo was rushed to market to take advantage of the ready pool of volunteers who had a bone to pick with Yahoo's link rot and limited scalability. Good move - before long, Newhoo had been acquired by Netscape for a tidy sum.

A key thing that the design of Newhoo failed to take into account - at least as far as its work force was concerned - is that, as Thomas puts it, "the ultimate directory of Internet content is a very valuable thing... indeed one of the most valuable things there is." Therefore, the "compensation model needed to be perfect," and a key part of the plan had to be to tie the payouts to the flow of incoming revenues. The thought of not paying editors at all didn't factor into Wherewithal's Searchonomics plan.

To push the Laffer Curve analogy perhaps a bit too far, imagine if you were employed as an editor in a hierarchical republic whose rate of taxation was effectively 100%. Editing activity, in the long run, might drop close to zero as incentive to perform services was greatly diminished, and as contributors opted for citizenship in a republic with a lower tax rate. Now imagine if the rate were pushed close to 0%. Advertising revenues? They're all yours! Great, you, as an editor, get to take home more loot, but the operation isn't fiscally sustainable. The guys who keep the lights on and the engines well oiled suddenly disappear, and you're back to complete anarchy. Wherewithal needed to find a way of conveying the right percentage of advertising revenues to its potential labor force of infomediaries.

Calling all humans

Wherewithal makes a compelling pitch to current Open Directory editors that they might consider upgrading from their less-than-sweatshop pay scale at ODP. In fact, a large part of Wherewithal's current strategy has involved targeting ODP editors, and even allowing them priority in signing up on Wherewithal's system so they can "reclaim" their old categories. This has led to some allegations that Wherewithal intends to make "unauthorized" use of ODP data in contravention to the ODP license. Wherewithal maintains that it is posting ODP attribution appropriately on its web site. ODP data is, indeed, used in different ways by different licensees, including the Google Directory, which doesn't provide ODP attribution on every page, but only insofar as the license requires it.

Thomas and his team have run some interesting tests of the Open Directory's claim to have 30,000 active category editors. In their analysis, there are only 9,000 ODP editors who "actually have a category," and only about 5,000 have done more than one or two edits. A central core of 500 do the bulk of the editing, many observers believe. While humans may do it better, there are fewer humans working for ODP than previously believed. Yahoo! and Looksmart have even fewer than 500.

Thomas believes that the Wherewithal system provides a more robust platform for editors to shine and to be paid well for linking users to the content they're seeking, particularly in targeted commercial sectors. Thus he believes that it will only require 2,000 editors to rival ODP's database in terms of quality. The hope, of course, is to find many more than 2,000.

Some kid with a 486 in the basement?

David Prenatt, Wherewithal's recently-hired Chief Evangelist and a former Open Directory Project editor (see " Life After the Open Directory Project"), was even more optimistic than his boss. "In three months," asserts Prenatt, "our database will make ODP look like some kid with a 486 in a basement."

The fact that editors may earn money for their toils - while important - is not the most impressive feature of Wherewithal. The scalability of the project is compelling. ODP claimed to solve the scale problem, but ran into problems of personnel administration as well as the problem of quality control that is bound to crop up in a volunteer project. Wherewithal, by contrast, allows anyone to own a category, and doesn't have to "replace" bad editors with good ones. End users can, in essence, set up their own customized directories by picking and choosing their preferred category owners. Wherewithal intends to play virtually no centralized role as an arbiter of good content or good editing. "Our model is to stay out of the way, provide the platform, and make sure the computers don't crash," summarizes Thomas. Spoken like a true advocate of laissez-faire.

Another plus associated with the Searchonomics system is its flexibility. Parts of the directory could be plugged into a vertical portal site. The ads could even be turned off if the site wanted to find its own volunteer editors or pay a higher fee for the use of the directory.

T minus $5 million, and counting

Of course, many of these features exist only at the design stage, and await full implementation of the project. Like Reaganomics or a rocket launch, the blueprint might look nice, but you never know how things will shake out until the mission is underway in real time, affected by the steering actions and interactions of those most unpredictable of actors, real people at multiple levels (end user, editor, engineer, site owner, ad rep, advertiser), and the economy they create. Wherewithal is an angel-funded startup with eleven staff, and is currently putting together its proposal for a Series A round of venture capital funding.

To attract the volume of end-user traffic that would generate the needed ad revenues to pay the category owners (one of several "chicken and egg" questions facing Wherewithal), it would seem clear that Wherewithal will, at some stage of its development, find itself portal infrastructure deals - to propagate itself far and wide as companies like Looksmart, Dmoz, and Quiver have sought to do. No traffic, no revenues. No revenues, no company.

Really big computers

Getting back to basics: will this help us find information? In Thomas' view, the sheer number of potential editors - tens of thousands or even millions - that can be built into the Wherewithal system without it cracking under the strain make it the ultimate Internet research tool. One indicator that this startup is dead serious about the scope of its ambitions is that Wherewithal already has *really big computers*. The company's web site contains an impressive description of the design of the search engine technology which powers the system, and the use of dynamic HTML and other means to improve speed. Wherewithal engineers reportedly sought out the number of searches performed on Yahoo! per second at peak times, added a zero, and went to work designing their system to handle a heavy load.

Listening to Thomas' careful description of the various operational and scientific principles behind the project, it's clear that this wasn't in the same league as many business models in the pay-you-to-surf realm, many of which were one-dimensional and unable to keep their promises. Indeed, from Day 1, the goal was to design a web search technology which would embrace the chaos and promise of the Internet. Wherewithal's hands-off, no-hierarchy approach would, paradoxically, offer a means of taming that chaos and fulfilling that promise.

From scribbles on napkins to an Internet revolution called Searchonomics? Stranger things have happened.


Can you afford not to know this stuff?
Sign up for the weekly Traffick Newsletter and stay informed!

Traffick Weekly
Are you the smartest Internet expert on the block? If not, you haven't been reading the Traffick Weekly!

The Traffick Weekly Newsletter is the fastest way to keep on top of key Internet trends. It offers a wide ranges of sources, and is edited to focus only on what you need. Enter your e-mail below to subscribe now, or read on to find out why the Traffick Weekly is a must-read!

Subscribe to the Weekly

Traffick Monthly
Stay up to date on the topics that are important to you. Subscribe to the monthly Traffick newsletter by entering your e-mail address below and clicking Join!

The monthly offers exclusive content and links to resources that you won't find anywhere else. Traffick is an Editor's Choice of Topica!

Subscribe to the Monthly

Portal Builders
Are you responsible for a portal project? Does your company provide portal building solutions? Just interested in issues related to building a portal? This discussion list will facilitate ongoing interaction amongst the community of professionals who are responsible for building portals.

Don't reinvent the wheel! Become acquainted with best practices by tapping into this valuable B2B resource. Portal solutions providers: you are welcome to introduce your company to the list. Others: please post your questions and comments to the list.

Subscribe to Portal Builders