Easy A/B test simulations using Excel / Google Docs
Simulated visitors, power, and significance on one page
Simulations can answer key questions without painful calculations. If you haven’t gotten around to learning R, here’s an A/B test simulator for Excel or Google Docs. It does a power calculation, so you can see the impact of baseline conversion rate, effect size, and traffic has on your chances of detecting an effect. It gives the effect size and p-value for each outcome.
Download Excel A/B test simulator
Here is the default simulation (every time you load it or refresh it, it’s different):
The default simulation settings and outcome
The default simulation shows an under-powered test. Here we have a baseline 5% rate, which we will increase by 10%. The sample is only 500 visitors, which gives ridiculously low power (aim for 80%). As expected, the result of the simulation is far from the true +10% effect. It’s actually a drop of 23.7% and we see this result is statistically inconclusive. The chart provided shows the chance streak on B which caused the variations to diverge:
Template includes a time chart
Learn what would happen if we added 14,500 more visitors.