Business hypotheses improve A/B tests
A hypothesis is an explanation of why something is the way it is.
Example business hypothesis:
“We are a new company, and visitors have doubts about the quality of our product.”
Do we really create hypotheses to test them?
To see if my example hypothesis is true, it would be best to talk to some potential customers. A/B testing is not really about testing business hypotheses but about using them to iterate a design. An A/B tester is not a scientist. He takes the hypothesis as inspiration for new visual treatments in order to increase his chances of raising revenue.
If we think that quality is a concern for visitors, we can add customer testimonials, quality certifications, and so on. And for each of these strategies, there are a million possible visual implementations. If we run an A/B test that shows a statistically convincing way to raise revenue, that’s a win, but it does not mean the business hypothesis that inspired it is actually true. Maybe testimonials drew more attention to the customer service, and quality was never the problem. If the test failed, that doesn’t disprove the hypothesis, because maybe a different way of doing testimonials would work.
Benefits: context, intentionality, and stronger concepts
If you test multiple visual ideas, some subtle and some more direct, you might get better insight into the business problem. But that’s not the primary benefit. Business hypotheses improve the process for generating visual ideas.
Business hypotheses are a way of challenging yourself and your team to justify visual changes and weed out implausible theories.
A hypothesis can be a theme that helps focus our efforts, makes it easier to organize to create and organize testing ideas. “Hey, I think our pricing might be confusing. Let’s attack that with a few tests next month”. Maybe there’s nothing wrong with pricing per se, but testing new ideas might reveal better ways of doing pricing.
Hypotheses also help reduce exposure to false positives. These days it’s easy to collect tons of data and then make numerous comparisons until you find a significant effect just by chance. In contrast, a well-reasoned hypothesis allows “prior information” to inform test results. For example, say we have some good reason to expect X to create a positive change in user behavior Y, and then we see this confirmed in a test. That result has a higher chance of being trustworthy compared to a random finding.
Give it more than once shot
It’s best not to give up on a business hypothesis too quickly, nor assume it’s true based on just one test result.
If it doesn’t work, try it another way.