Scientists test hypotheses, A/B testers use hypotheses
A hypothesis is an explanation of why something is the way it is.
Example business hypothesis:
“We are a new company, and visitors have doubts about the quality of our product.” It’s practically impossible to test this hypothesis with an A/B test. A/B testers test visual treatments, not business hypotheses.
Is it true? vs. Assuming it’s true, how can we use it?
A scientist designs an experiment to challenge the hypothesis and see if it holds up. This methodical, very precise experiment may have no immediate business benefit.
An A/B tester takes the hypothesis as inspiration and builds on it to create a new design. The purpose of testing the new design is to confirm that the new visual treatment has immediate business benefit. Even if successful, such a test does not mean that the business hypothesis is actually true.
Hypotheses provide context, encourage intentionality, lead to stronger concepts
This doesn’t always mean writing formal-sounding statements. It means challenging yourself and your team to justify visual changes and weed out implausible theories.
A hypothesis is a theme that helps focus our efforts. Stating explicit hypotheses can facilitate organization and ideation of visual treatments. “Hey, I think our pricing might be confusing. Let’s attack that with a few tests next month”. Whether pricing truly is confusing or not, testing new ideas might reveal better ways of doing pricing.
Hypotheses help reduce exposure to false positives. These days it’s easy to collect tons of data and then make numerous comparisons until you find a significant effect just by chance. In contrast, a well-reasoned hypothesis allows “prior information” to inform test results. For example, say we have some good reason to expect X to create a positive change in user behavior Y, and then we see this confirmed in a test. That result has a higher chance of being trustworthy compared to a random finding.
Null/alternative hypotheses are not business hypotheses
A null hypothesis for experimental purposes says that “there is no difference between the things compared”, while the alternative hypothesis is that “there is a difference”. But statistical hypotheses have nothing to do with the business hypothesis.
If you run a test and get a statistically significant result, you have indirect evidence whether a visual treatment caused an effect, but you have no evidence about your business hypothesis, which may be true or false, who knows.
Same business hypothesis, a million visual implementations
If we think that quality is a concern for our visitors, we can add customer testimonials, quality certifications, manufacturing details, reframe “new company” as “innovative”, and so on. For each of these many strategies, there are a million possible visual implementations.
A/B tests are so multi-faceted that they rarely if ever provide compelling evidence regarding a business hypothesis. If we tried customer testimonials and the test was a success, did we prove the hypothesis? No. Maybe testimonials drew more attention to the customer service, and quality was never the problem. If the test failed, do we reject the hypothesis? No. Maybe a different way of doing testimonials would work.
If a test fails, we should not give up on a business hypothesis too quickly. Even though an A/B test is usually insufficient to test a business hypothesis, if you attack it from multiple angles, both subtle and more direct, you might get better insight into the business hypothesis. You should test important business hypotheses with more direct methods like user research, surveys, and so on.