How important is it to avoid A/B test collisions?

Question

Let's say you have 3 A/B tests as such

1. Control vs A
2. Control vs B
3. Control vs C

Unless you specifically work to ensure that tests do not collide (run on top of each other), you will end up with populations of users who see both A+B, B+C, A+B+C.

Technically, because if you're looking at results of experiment 1, both Control and A will see an equal number of pollution from 2 and 3, so the only difference in performance between 1. Control and A should be just the impacts of A.

Thoughts? Is it always best practice to avoid collisions or can we just assume, so long as pollution is evenly distributed, we're good?

Answer 1

Generally this should be done as a series of isolated A/B tests or as a multivariate test.

It's hard to say for certain without more details, but it appears that you have 4 possible versions of your page and you want to see which one performs better.

You could compare the control vs. A as one test, and see which one wins. Then, pit the winner of that test against B, and then the winner of that against C. (If the control loses, you don't want to waste time testing it against B or C.) You may want to leave some time between tests for the page to "settle" so users aren't always seeing new stuff on the page every visit. However, this plan means that you could wind up with slightly different user populations participating in each test over time. (For example, if version A has lots of kittens and wins you may attract more cat lovers to the site. Then, if version C has lots of puppies it may not score as well as if it had gone first.)

To compensate for that, you could run this as a multivariate test where you have 4 options: control, A, B, & C. 25% of your traffic is sent to each one and they all compete simultaneously to see which version performs better. The downsides are that you have to have sufficient traffic to split it 4 ways, and users and your support team may become confused seeing different versions of the site.

This brings us back to collisions. The common way to track A/B tests is with cookies. Each user's randomly assigned version is remembered, so when they come back they still see the same page. However, it is very common for users to clear their cookies or check sites on their phones or work computers resulting in them seeing a different version.

Depending on the scale of your changes, this is probably OK from an experience perspective, but could be messy from a data quality point of view. Users probably won't be too concerned by small changes like button color or label changes, but you won't strictly know if seeing version A on their phone caused them to transact on version B at work or if B did all the influencing.

There are more invasive or complicated ways to track things, but in general the only way to avoid the issue is to run the test on the smallest population for the shortest amount of time needed to produce a statistically valid result.