Split-testing Ads on Google

[Kirk]

I wrote this little app about a year ago when I first started split-testing my Google ads:



I had found a few existing tools that did the same thing, but they used differences in standard deviations to calculate the result. I asked my wife about it (statistician who runs clinical trials) and she recommended the chi-squared test as a superior method to determine statistical significance.

I dug the formula out of an old stats book (that class still brings back bad memories...) and coded this up.

Please add any recommendations on how I might improve this. I find it works well and automatically adjusts the significance on sample sizes less than 30...

[Gerardas Norkus]

Hi Kirk,

I know very similar site to yours for checking out split testing results at:

http://www.splittester.com

I hope this will help you to improve yours.

[Kirk]

Thanks Gerardas,

This site was actually the inspiration for my swing at this type of application. I heard about it on a Perry Marshall call and tried it out on a few campaigns I was running. Although it seemed to work, it didn't mention anywhere about how the significance was derived. I asked my wife about it and she pointed me to the chi squared statistics test commonly used in clinical trials.

I setup the chi squared formula in Excel, but couldn't replicate the results achieved at splittester.com. To ensure I had built the formula correctly, I located a few chi squared calculators at some university websites - everything checked out. I then just transferred my excel worksheet to PHP code and ran the same tests.

While I'm not saying the results at splittester.com are incorrect, I was just uncomfortable with the lack of detail on how the results were derived.

Truth be told, I was learning PHP at the time and was really just looking for an excuse to do a little coding in it.

[Gerardas Norkus]

Thanks Kirk,

Well, I'll try your script as soon as I need it and compare it to splittester.com

[nuttymarketer]

Dear Kirk,

I do not see any point in using splittester.com This is a children toy.....

Please correct me if I am wrong.

Arvind Kumar

[Kirk]

[nuttymarketer]

Dear Kirk,

Thanks for that. I would like to see the point. Please explain it in detail.

Thanks in advance.

Arvind

[Kirk]

I got it. Let's say you write an ad and it performs with a CTR of 1%. You get an idea for a new ad that you think will beat your existing ad (since you've only tried one ad so far, that's called your "control" ad).

You decide to delete your existing ad (the "control") and replace it with the new ad you've written. After a couple of hours, your new ad is performing at a CTR of 1.3% - you think you've beaten the control ad, but there's one small problem...

The problem is, you don't know if the new ad is actually better because the results might be due to random chance. A good example is flipping a coin 5 times - you might get "heads" five times in a row because of random chance. If you flip the coin 100 times, you'll get heads 50% of the time - not 100%.

So my tool, and the tool at splittester.com, determines through statistical methods what is random chance and what is not. My tool offers a statistical significance which lets you gauge how much uncertainly has been factored out - 80% confidence is good, but 99% confidence is better...

Google adwords offers a great feature which allows you "test" multiple ads simultaneously - it will serve up each ad randomly and track the impressions and clickthroughs. You just plug those stats into my tool to get your answer...

I hope that helps.

[nhynes57]

Kirk,

That is a useful tool. I never realised that there might be a such a difference,
but then I could never get my head around that statistical stuff anyway.

Noel