Should you ever test multiple A/B tests at the same time?

Question

Okay, so you have a website with lots of web pages on it. It has a header/nav and a footer that show up on every page, obviously. And you have a bunch of other unique individual pages.

You want to run some A/B tests on several different pages on the site. However, we also want to run some A/B tests on the navigation. The navigation that is on every page...

So, if you take some random tested page: you have different test variations of this page as well as different test variations of the global navigation on this page as well.

So how do you determine test winners when you have multiple tests at once? If a visitor is seeing Nav variation B and page variation B, how do you know which test is actually the one contributing to the "increase in conversions" (or whatever your success metric is)?

Alternatively, you could just split your traffic so that people are only in one test a time. It just seems like you are potentially limiting your audience (depending on how much traffic your site gets) if you have a lot of different tests on the site you want to run.

Or would another alternative be that whenever you need to test a global ui element (like a nav or footer) that you cease all other individual, one-off testing on the site? That seems like that would work but it really limits how much you can test during that time period.

What is the best approach to this problem? What is the best practice for deploying multiple tests on a site at the same time?

Answer 1

Multiple A/B tests not only are very common, but they are extremely useful, way more than simple A/B. However, it has a different name: Multivariate Testing.

Just imagine this: I run a test for page A and page B and decide page B performs better than page A. Great, let's go with page B. Easy, no? Then I decide to test another element, say a button, and button A is better than button B. So I use button A on page B and problem solved.

Now, the question is.... is button A performing better on any instance, or if I test button A on page A it will work better? So... back to point 0, I have to test everything again instead of doing it all at once!

Multivariate to the rescue!

Instead of doing something like the case above, try multivariate testing (or multiple A/B)

Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.

Websites and mobile apps are made of combinations of changeable elements. A multivariate test will change multiple elements, like changing a picture and headline at the same time. Three variations of the image and two variations of the headline are combined to create six versions of the content, which are tested concurrently to find the winning variation.

The total number of variations in a multivariate test will always be:

[# of Variations on Element A] X [# of Variations on Element B] ... =
[Total # of Variations]

Remember, context is everything, so nothing works like in a clean lab. You'll have multiple variables, and the example above is an extremely basic one, let alone if you add features, consider geo, devices, time of the year, audience age and so on.

So, in short: as long as you can define a reasonable subset and test them as much as possible go with multivariate testing

Answer 2

There is no need to avoid it, and MVT tests are not a solution, since there is no conceptual difference between A/B and A/B/n tests.

Still, there are some caveats you should be aware of, since there is a theoretical possibility that the tests will influence each other in a way that results in wrong conclusions from at least one of the tests. You might want to read this for a more detailed analysis of the potential issues: http://blog.analytics-toolkit.com/2017/running-multiple-concurrent-ab-tests/