vlad malik

Easy A/B test simulations using Excel / Google Docs
2 years ago

Simulated visitors, power, and significance on one page

Simulations can answer key questions without painful calculations. If you haven’t gotten around to learning R, here’s an A/B test simulator for Excel or Google Docs. It does a power calculation, so you can see the impact of baseline conversion rate, effect size, and traffic has on your chances of detecting an effect. It gives the effect size and p-value for each outcome.

Download Excel A/B test simulator

Example

Here is the default simulation (every time you load it or refresh it, it’s different):

000964

The default simulation settings and outcome

The default simulation shows an under-powered test. Here we have a baseline 5% rate, which we will increase by 10%. The sample is only 500 visitors, which gives ridiculously low power (aim for 80%). As expected, the result of the simulation is far from the true +10% effect. It’s actually a drop of 23.7% and we see this result is statistically inconclusive. The chart provided shows the chance streak on B which caused the variations to diverge:

Simulation_AB_chart

Template includes a time chart

Learn what would happen if we added 14,500 more visitors.

@VladMalik is an interaction designer and musician based in Toronto.
I enjoy breath-hold diving, weight-lifting, and chopping wood. I am vegan.

Michael wrote on March 15, 2017

Hi Vlad, i noticed the Easy A/B test simulator link is broken regards Michael

Reply

Share your thoughts!

© 2015 License for all content: Attribution not required. No commercial use.