Everyone knows what a great job the big boys do. Amazon, eBay--you can go down the list. All of these "born on the web" companies have massive feedback loops that allow them to experiment with small changes in their marketing, their user experience, their offers, and ten other things. Whenever they make a change, they see if more people are buying and decide whether the new way "works" or not. Amazon's CTO once told me they can change a font at breakfast and know whether it was a good idea by lunchtime. But how can you do the same thing, when you don't get nearly as many visitors as these big websites do?

In a sense, you are worried about something important. Because these sites have so many visitors, they can make a decision about their change much faster than you can about yours. And they can be more sure that they are right than you can. 

So, you could throw up your hands, saying, "Unless our statistics reach statistical significance at the 95th percentile, we aren't going to make decisions on them." Or you could say, "We'll never have a large enough sample size to make up our minds." 

Or you could take a different approach. 

One is to wait longer. Accept the fact that if you get 1,000 visitors a month instead of 1,000 a minute that it will take you longer to evaluate your changes than Amazon. And that can be OK. If you have a small site, you probably don't want to spend all your time running tests and making changes anyway--you have the rest of your business to run.

But there is another approach that you take in other areas that works here, too. Don't worry so much about being absolutely sure of the answer. If three people walked into your store and each decided against buying the sale item you had displayed, even after they looked it over carefully, would you leave that display as is until 20 more people did the same thing just to be sure it was a clunker?

Probably not. You'd start to tinker with it, even though it's entirely possible that those people were not representative of the population at large.

You can do the same thing with your website and your Internet marketing. If you have a page that seven out of eight abandon your site from, you can try to make a change to that page and see if you do better with the next eight people. It is very possible by taking this approach that you will sometimes act on unreliable information, but it is very unlikely that you'll have that kind of bad luck many times in a row. By experimenting more, you are likely to move in the right direction.

So don't let low visitor counts stop you from experimenting. Remember that if you don't look at your statistics at all, you are experimenting anyway. You just never know what is working. At least this way, you know what is working some of the time--perhaps even most of the time.

October 16, 2013

Mike is an expert in search marketing, search technology, social media, publishing, text analytics, and web metrics, who regularly makes speaking appearances.

Mike's previous appearances include Text Analytics World, Rutgers Business School, SEMRush webinar, ClickZ Live.

Mike also founded and writes for Biznology, is the co-author of Outside-In Marketing (with James Mathewson) and the best-selling Search Engine Marketing, Inc. (now in its 3rd edition, and sole author of Do It Wrong Quickly, named by the Miami Herald as one of the 11 best business books of 2007.


I think no. From the word itself, EXPERIMENT, you must not expect anything. I mean it is unpredictable. To exert a pull on more or fewer customer will be the upshot of your Experimental Marketing. It would be your basis if the service or product that you are testing will click or not.

Well, for you to consume less of a time. Two or several plans must be made during the experimental marketing day.

Nice post and very helpful!

In the news aggregator and social bookmark site for Internet marketers - kingged.com, the above comment was left.

Hi, very informative post there ! I am particularly in agreement with the fact that Internet marketing based on experimentation can actually help in the long run. There's certainly a law of average working there whereby you can hardly fail for a prolonged period of time!

"So don't let low visitor counts stop you from experimenting."
I disagree.
For example, the first visitor is your Mom, and she likes your site very much. You'll have the feedback, that 100% cof your visitors are satisfied.
I think, you have to wait for several hundred/thousand visitors, to have any useful data about the results of your experiment.

It's very hard to have patience to wait for enough data after a change. A beginner small business owner have to wait for weeks to have any idea, whether a new method/approach/campaign works or not.
It's tempting to make little changes every day, but you won't get real feedback because the low amount of visitors.

Sounds like we actually agree. i said in the post that waiting was the most obvious way to do this. I don't think you disagree--you don't think that small businesses should avoid experimentation. I am not saying you must make changes each day, but you shouldn't be afraid to.

Just because you are a small business doesn't mean that your not able to make decision based on the information you have. You need to take decisions in order to get more visitors to get better data and make more relevant decisions. Someone once explained to me that from little data you can sometimes make up more that is relevant to you. Meaning yes 7 out of 8 people dislike your site but you can find out that from those 8, 3 came from a query from Google, 3 came from sites that were not related to your topic, 2 used direct links. So basically 3 out those 8 were irrelevant for your overall statics. Great article :D

You can tinker with your site if 8 persons come on your site and none of them buy or you can observe what the big boys have already done with their site since they have done extensive testing already and just do what they are doing and use their template as the foundation for your website.

I think, that you will have to have patience if you want to get something from your experimenting. Of course you can experiment with any website, but know just what kind of result you are looking for. Is it higher conversion rates or is it more time spent reading articles, getting likes and so on...

Sure the big boys can do almost whatever they want, because small changes wont affect their brand, the trust is already build to their brand. Small sites needs to experiment with factors that will build that trust!

Really great post!

Comments closed after 30 days to combat spam.

Search Engine Guide > Mike Moran > Does experimental marketing require lots of customers?