Those of you who know me realize that I am a big fan of feedback. The very title of my book Do It Wrong Quickly is a snarky way of saying that most of what we do is wrong, and that only by looking for feedback from customers can we realize what is wrong and what actually worked. So, I should be very excited about the announcement today of Yahoo!'s new analytics tool. And mostly, I am. But I am also starting to wonder when fast gets fast enough.

For those of you not following every breaking story on the newswire—what were you doing, anyway, running your business?—Yahoo! acquired a company a few months ago named Indextools, which makes a nice Web analytics engine. Today, Yahoo! announced that it's making it available in a private beta for my favorite price, free.

And that's good. Yes, we already had Google Analytics for free, but competition might keep Google innovating, which would be good. And Yahoo!'s offering might be better than Google's, which would be very good.

But in a quick scan of today's announcement, what jumped out at me was the claim for "real-time data." Instead of waiting 24 or 48 hours to get your data, it shows up in a couple of hours. And that's good, too. After all, who wouldn't want to get their feedback faster?

But I worry that faster feedback might substitute for smarter analysis. Most small businesses don't get enough feedback to decide anything in a few hours. They need days or even weeks of metrics before they can see whether a page change made things better or worse. They just don't get enough traffic to make decisions any faster.

But faster feedback is exciting. It's sexy. It gives us things to talk about. The problem is that what we need is more thinking about the numbers. More tests rather than faster tests. Most small businesses aren't testing enough things as it is—it won't make much difference to get the results faster.

Now, none of this makes Yahoo!'s new tool a problem. It will be a while before many of us get a chance to try it, but I expect it to be very good. But when do we realize that rather than expecting spiffy new tools to rescue us, that instead we must do the hard work of thinking? We need to be willing to make our decisions based on metrics, which means designing alternatives, deciding what are expected outcomes are, and trying them out to see what works better.

Real-time data is great, but it's no substitute for real analysis.