Searchonomics: A New Era in Human-Guided Web Search?
By Andrew Goodman (from Traffick Monthly newsletter) - August 29, 2000
Legend has it that a major plank in Ronald Reagan's
economic platform - the Laffer Curve - was sketched out
for him on a napkin. On this foundation, Reagonomics became
the ovverriding theme of a decade of US fiscal policy.
A vast array of reforms flowed from some basic reasoning that
essentially argued that if you put a luxury tax on boats,
sales will decline, and the luxury boat industry will dry up.
A lot of powerful ideas would appear to originate
this simply. 1998 was a year in which everyone in the Internet
business decided that there was room to improve on Yahoo! - in particular,
its directory, which, critics charged, couldn't scale with the growth
of Internet content. The overworked team of Yahoo! editors let "link
rot" set in, and ignored many submissions.
Can't anyone be a "surfer"?
It's telling that Yahoo!'s job descriptions refer
to directory editors as "surfers." If you're thinking
"just about anyone could do that," you're not far from wrong. And
in fact, there is a huge thirst amongst heavy Internet users to
get more involved in rating sites, offering advice, publishing articles, and
ranking products. A host of innovative business models are appearing
on the scene today, promising to give a platform, and sometimes
some pecuniary rewards, to these legions of enthusiasts and experts.
This new type of knowledge worker may potentially fuel the next
leap forward in information delivery on the Internet.
One team of inventors, led by Steve Thomas, a
software developer and aficionado of cognitive science, sketched
out some principles for the ultimate web search resource "in a one-page concept
paper" in February of 1998. Barely larger than the proverbial
napkin, this paper kicked off a process of planning and development
for a new system of "Searchonomics" under the company name Wherewithal.
Thomas had been Netscape's Product Manager, Platform
called "layers." With Netscape since shortly after its IPO
in 1995, he left in 1998 to start Wherewithal. Along with co-founder
and CIO Darren Skinner, Thomas sketched out key principles
for a new web search resource to end all resources:
- Scalability. Observing Yahoo!
and other human-managed directories, Thomas and his team believed
that most of them were designed with assumptions of the "small
scale Internet" in mind. After all, Yahoo! began as Jerry Yang's
Guide to WWW, a helpful guide to Internet content in different
subject areas. It wasn't all that hard to build. Set up an arbitrary
taxonomy, get a few people working on it, and reasonable coverage
of the web was possible. As the Internet expanded, this model,
in Wherewithal's view, became obsolete.
- Human element. One thing Yahoo! did
have going for it was the judgment employed by category editors.
Robots can be inflexible, and do a poor job of acting as
"gatekeepers" and judges of what's good and what isn't. The raw
knowledge available on the Internet needs to be digested by "informediaries"
if it is to be useful.
- The possibility of multiple taxonomies.
If you think about a Yahoo! category, or for that matter,
the category structure in any human-edited directory or web guide
(eg. Looksmart, Open Directory, or About), the choice of how to
organize the resources in a given field - which to highlight,
which to downplay, which sites to accept, and which to reject
- is made by a single editor. Even if there is a committee system
in place in an editorial organization, the end user only sees
one category structure and one opinion as to what resources about sports
cars (for example) are most useful. Such an arbitrary method
for a medium which is supposed to be about flexibility,
self-publishing, even chaos! The Wherewithal plan needed
to come up with a way for the end user to substitute their favorite
editor (a so-called Second Opinion) in place of one which they
felt wasn't suitable for their needs. This builds accountability
into the system. There is no such thing as a "bad" editor (the
users will determine that), and therefore command-and-control
administration of the editorial personnel isn't needed.
- A sustainable incentive system.
Why would anyone be a category owner in the first place? The
work of infomediaries has value. Over time, some analysts predict,
the compensation of such infomediaries will adjust to closer
reflect the value of that work - in part, through choices made
by the infomediaries themselves. We're well past the days
when we simply volunteered advice in Usenet newsgroups without
any expectation of reward. The act of volunteering advice
or pointing people towards useful resources isn't going to
go away, but most infomediaries and consultants today have
a sharper sense of how soon to put their "clients" on the
"meter." In a manner similar to About,
which pays its guides a percentage of the revenues generated from
their About.com Guide Sites, Wherewithal's Searchonomics concept
pays editors a portion of the revenues from targeted advertising
in their directory categories. Editors who attract more users
through the high quality of their work, or simply through the
act of editing in more marketable categories, stand to make more
Newhoo Built a Valuable
As Thomas tells it, Wherewithal became aware of
the advent of Newhoo (now Open Directory)
about six months into the life of their own project. Wherewithal
was diligently working on the back end - the technical and
operational system which would give the project legs. By contrast,
Newhoo was rushed to market to take advantage of the ready pool
of volunteers who had a bone to pick with Yahoo's link rot and limited
scalability. Good move - before long, Newhoo had been acquired by
Netscape for a tidy sum.
A key thing that the design of Newhoo failed
to take into account - at least as far as its work force was concerned
- is that, as Thomas puts it, "the ultimate directory of Internet
content is a very valuable thing... indeed one of the most valuable
things there is." Therefore, the "compensation model needed to be
perfect," and a key part of the plan had to be to tie the payouts
to the flow of incoming revenues. The thought of not paying editors
at all didn't factor into Wherewithal's Searchonomics
To push the Laffer Curve analogy perhaps a bit
too far, imagine if you were employed as an editor in a hierarchical
republic whose rate of taxation was effectively 100%. Editing activity,
in the long run, might drop close to zero as incentive to perform
services was greatly diminished, and as contributors opted
for citizenship in a republic with a lower tax rate. Now imagine
if the rate were pushed close to 0%. Advertising revenues? They're
all yours! Great, you, as an editor, get to take home
more loot, but the operation isn't fiscally sustainable. The guys
who keep the lights on and the engines well oiled suddenly disappear,
and you're back to complete anarchy. Wherewithal needed
to find a way of conveying the right percentage of advertising revenues
to its potential labor force of infomediaries.
Calling all humans
Wherewithal makes a compelling pitch to current
Open Directory editors that they might consider upgrading from their
less-than-sweatshop pay scale at ODP. In fact, a large part
of Wherewithal's current strategy has involved targeting ODP editors,
and even allowing them priority in signing up on Wherewithal's system
so they can "reclaim" their old categories. This has led to some
allegations that Wherewithal intends to make "unauthorized" use
of ODP data in contravention to the ODP license. Wherewithal maintains
that it is posting ODP attribution appropriately on its web site.
ODP data is, indeed, used in different ways by different licensees,
including the Google Directory, which doesn't provide ODP attribution
on every page, but only insofar as the license requires it.
Thomas and his team have run some interesting
tests of the Open Directory's claim to have 30,000 active category
editors. In their analysis, there are only 9,000 ODP editors who
"actually have a category," and only about 5,000 have done more
than one or two edits. A central core of 500 do the bulk of the
editing, many observers believe. While humans may do it better,
there are fewer humans working for ODP than previously believed.
Yahoo! and Looksmart have even fewer than 500.
Thomas believes that the Wherewithal system provides
a more robust platform for editors to shine and to be paid well
for linking users to the content they're seeking, particularly in
targeted commercial sectors. Thus he believes that it will only
require 2,000 editors to rival ODP's database in terms of quality.
The hope, of course, is to find many more than 2,000.
Some kid with a 486 in
David Prenatt, Wherewithal's recently-hired Chief
Evangelist and a former Open Directory Project editor (see "
Life After the Open Directory Project"), was even more
optimistic than his boss. "In three months," asserts Prenatt, "our
database will make ODP look like some kid with a 486 in a basement."
The fact that editors may earn money for their
toils - while important - is not the most impressive feature of
Wherewithal. The scalability of the project is compelling. ODP claimed
to solve the scale problem, but ran into problems of personnel administration
as well as the problem of quality control that is bound to crop
up in a volunteer project. Wherewithal, by contrast, allows anyone
to own a category, and doesn't have to "replace" bad editors with
good ones. End users can, in essence, set up their own customized
directories by picking and choosing their preferred category owners.
Wherewithal intends to play virtually no centralized role
as an arbiter of good content or good editing. "Our model is to
stay out of the way, provide the platform, and make sure the computers
don't crash," summarizes Thomas. Spoken like a true advocate of
Another plus associated with the Searchonomics
system is its flexibility. Parts of the directory could be
plugged into a vertical portal site. The ads could even be turned
off if the site wanted to find its own volunteer editors or pay
a higher fee for the use of the directory.
T minus $5 million, and
Of course, many of these features exist only
at the design stage, and await full implementation of the project.
Like Reaganomics or a rocket launch, the blueprint might look nice,
but you never know how things will shake out until the mission is
underway in real time, affected by the steering actions and interactions
of those most unpredictable of actors, real people at multiple levels
(end user, editor, engineer, site owner, ad rep, advertiser), and
the economy they create. Wherewithal is an angel-funded startup
with eleven staff, and is currently putting together its proposal
for a Series A round of venture capital funding.
To attract the volume of end-user traffic that
would generate the needed ad revenues to pay the category owners
(one of several "chicken and egg" questions facing Wherewithal),
it would seem clear that Wherewithal will, at some stage of
its development, find itself portal infrastructure deals - to propagate
itself far and wide as companies like Looksmart, Dmoz, and Quiver
have sought to do. No traffic, no revenues. No revenues, no company.
Really big computers
Getting back to basics: will this help us
find information? In Thomas' view, the sheer number of potential
editors - tens of thousands or even millions - that can be built
into the Wherewithal system without it cracking under the strain
make it the ultimate Internet research tool. One indicator that
this startup is dead serious about the scope of its ambitions is
that Wherewithal already has *really big computers*. The company's
web site contains an impressive description of the design of
the search engine technology which powers the system, and the use
of dynamic HTML and other means to improve speed. Wherewithal engineers
reportedly sought out the number of searches performed on Yahoo!
per second at peak times, added a zero, and went to work designing
their system to handle a heavy load.
Listening to Thomas' careful description
of the various operational and scientific principles behind the
project, it's clear that this wasn't in the same league as many
business models in the pay-you-to-surf
realm, many of which were one-dimensional and unable to keep their
promises. Indeed, from Day 1, the goal was to design a web
search technology which would embrace the chaos and promise
of the Internet. Wherewithal's hands-off, no-hierarchy approach
would, paradoxically, offer a means of taming that chaos and
fulfilling that promise.
From scribbles on napkins to an Internet revolution
called Searchonomics? Stranger things have happened.
Can you afford
not to know this stuff?
up for the weekly Traffick Newsletter
and stay informed!