I was taught when A/B testing every test should be based on a test hypothesis, e.g.

Control: Current Button (Blue) Test: White Button

"We believe that changing the CTA colour (British English spelling!) from Blue to White will increase Click rate because the button will be more noticeable due to the increased contrast ratio of white on our default black background. We will know this to be true when we see an increase in CTR % on the homepage to 95% statistical confidence."

Should I be writing a test hypothesis for a multivariate test. I want to find the optimum combination of button color, label and icon to increase Click Rate?

2 Answers 2


In general, any kind of test and research is supposed to have an hypothesis. I won't say ALL kinds because nowadays you've automated tests created by machines using machine learning. But in general, the answer is YES, you should have a hypothesys on A/B as well as multivariate.

However, on this kind of tests (specially A/B) the hypothesis is usually "it will work better". Better engagement, better CTR, better whatever.

So, in practice, most of us just write the change to do and that's it. It's more important to document the changes than to document the hypothesis, because you'll probably go through many changes, and the hypothesis will always be the same.

In short, to answer your specific question: YES. However, is the least important part of the test.

If you want to learn more, I have wrote an article in 2 parts (one for A/B and one for multivariate) which you can find at A/B and Multivariate Testing. What are they?. I wrote these articles in Spanish some time ago, but translated them to English. I think you'll find some useful tips on how to work with both A/B and Multivariate

  • Your article link simply links to this question o.0
    – straya
    Feb 10, 2020 at 1:32
  • Thanks, this is great. I will read the article (both parts) this morning Feb 10, 2020 at 10:54
  • Sorry, the link links back to the original question. What's the url please? Feb 10, 2020 at 10:56
  • sorry, fixed the link now :)
    – Devin
    Feb 10, 2020 at 17:47

Yes you should. Note that unless you have a gazillion users, it is often better to do A/B testing anyways, e.g. if you have A...E variations, first test:

A/B --> B wins!
B/C --> B wins!
B/D --> D wins!
D/E --> D wins!

This way, tests finish faster, you learn faster, and the design delivers value sooner.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.