First, we took a look at types of ads that you can test and then we look at how to analyze the results of your tests and set up new ones. But, one more thing we must know is when to consider a test complete and ready to be analyzed. If you consider a test complete before you have statistically significant results that prove with great confidence that what you observe is actually true, you may find yourself making conclusions that simply are not. Therefore, you need to know when you have enough data for this to be the case.

You've got two options -

1. Use the chi-square or analysis of variance methods in Excel. You will have to do some studying to figure this one out, but it's well worth the time.
2. Follow some basic rules for how much data you need before you can draw conclusions.

You'll have to dive into #1 on your own. Let's discuss #2 a little bit. Here are two major guidelines to follow when deciding when to end a test -

  • Capture data over the entire span of your buying funnel - What good does it do to get data for the first half of a buying cycle? You don't want to capture just your "one night stand" users; those that buy on the first click to your site. You want to make sure you collect data long enough to capture a good sample from every phase of the buying funnel from awareness to purchase. Run the test for like 3 sales cycles to get a good sampling.
  • Have enough traffic that one click or conversion will not significantly change your results - 500 clicks and 10-20 conversions are good benchmarks here. If your account doesn't get lots of clicks, you can spread out your ads across ad groups and analyze ad performance across multiple ad groups and keywords. You might write 4 different ads and run them on all of the ad groups in a particular campaign while just changing the headline to contain your ad group's theme keyword in it. Doing this allows you to still compare different ad messages against each other to find out what's important to your target audience.

The situation you don't want to run into here is where the next click or conversion changes the entire scope of your test. For example, if you had 300 impressions and got 10 clicks for a 3% CTR and someone clicked on the next impression that would take you up to 3.7%. That's too big of a jump. Similarly, if you had 2 conversions in those 10 clicks for a 20% conversion rate and that next click didn't convert, that takes you down to 18%. Also too big of a jump.

So, this is actually a great test to decide if you have enough data. Calculate how getting a click on your next impression and a conversion on your next click affects your numbers. If there isn't a significant change, your test is over.

To summarize, if you let your ads run through all of the possible conditions that they could (weekdays, weekends, all hours of the day, all stages of the buying funnel, etc.) and you have enough data, then you are ready to analyze it.
July 2, 2010

Mike Fleming specializes in Analytics and Paid Search for Pole Position Marketing, a leading search engine optimization and marketing firm helping businesses grow since 1998. You can follow Mike on Twitter at @SEMFlem. Mike enjoys playing, writing and recording music along with playing basketball to get his workout in. He resides in Canton, Ohio with a girl who threw a snowball at him one day…then married him.

Mike and the team at Pole Position are available to help clients expand their online presence and grow their businesses. Contact them via their site or by phone at 866-685-3374.


For the start, we need money. That's the thing where I struggle at the moment, getting money for initial testing. I would really like to see a version of this series regarding affiliate marketing since you can't really measure everything with it.

Best wishes,

Great article Mike - i read the other two and didn't comment but after seeing a few more of your incite's in this area i had to just stop and say thanks for the information!


Comments closed after 30 days to combat spam.

Search Engine Guide > Mike Fleming > Use Ad Testing to Help Reach Your PPC Marketing Goals - Part 3