Is fair comparison a must in AB testing?

Question

I wonder if the elements in AB test need to be exactly the same so as to allow a fair comparison?

I found it personally depending on different scenarios.

For example if it's a AB test in smaller scale (e.g. a module to recommend users with similar products on product details page) and concrete goal (e.g. testing what layout drives a higher AOV) with 1 control and 2 variants of layout & placement - each of the product cards in this module contains price (sale price and original price), product name and image. I tend to make sure the elements are used across the variants (e.g. if there is a discount price, all variants should have it)

While if it's a test involves more of a conceptual thought or hypothesis (e.g. airbnb tests multiple variants of homepage simultaneously) and I found the variants can be executed quite differently (e.g. one concept can focus on room search vs the other being inspiration driven and etc) It's technically hard for every elements or variables to be the same and comparable and need further drill down of reasons for why it's winning even if there is a winning variant.

Wonder someone can share their experiences and thoughts on this?

Answer 1

No, you can test different sizes as long as they're the only variable you measure. Otherwise, if changes go further (including the fact that a different size of 1 element could mean different variables for all other elements), you'll need to go with multivariate , probably using MANOVA. An easy example: if your size change makes other content go over or below the fold, there's a high chance you'll need to consider MVT

In short: as long as the changes are on one element and they don't affect other elements in a considerable way, then you can go for it with plain A/B

Answer 2

Is fair comparison a must in AB testing?

No, providing the different versions facilitate the same goal, you can theoretically test very different versions, but the quality of the information you gain (and hence the quality of the test) deteriorates rapidly the more different the versions are.

Usually we with AB testing, we're looking at the affect of changing on specific element (having two more versions of that element).

But there isn't any reason you can't test two almost completely different e.g. landing pages, provided they both have the same goal.

The diversity of page types on the Airbnb site can be tested providing the goal is to increase click-through to certain pages, or some other shared goal, though in terms of gaining precise and repeatable knowledge of which factors are actually affecting conversion, a test like this is close to useless.