1

Okay, so you have a website with lots of web pages on it. It has a header/nav and a footer that show up on every page, obviously. And you have a bunch of other unique individual pages.

You want to run some A/B tests on several different pages on the site. However, we also want to run some A/B tests on the navigation. The navigation that is on every page...

So, if you take some random tested page: you have different test variations of this page as well as different test variations of the global navigation on this page as well.

So how do you determine test winners when you have multiple tests at once? If a visitor is seeing Nav variation B and page variation B, how do you know which test is actually the one contributing to the "increase in conversions" (or whatever your success metric is)?

Alternatively, you could just split your traffic so that people are only in one test a time. It just seems like you are potentially limiting your audience (depending on how much traffic your site gets) if you have a lot of different tests on the site you want to run.

Or would another alternative be that whenever you need to test a global ui element (like a nav or footer) that you cease all other individual, one-off testing on the site? That seems like that would work but it really limits how much you can test during that time period.

What is the best approach to this problem? What is the best practice for deploying multiple tests on a site at the same time?

  • Perhaps you could ensure that the navigation A/B test only runs on variation A of the pages (probably the current version, if applicable). That seems to me like it'd eliminate any doubt about the results, but I don't have a lot of experience in this area so I'm intrigued to see what kind of answers you get. – Nate Green Aug 1 '16 at 17:14
  • You could also just compare subsets of tests. Say User 1 in in the control for Test #1 and in the test for Test #2. You could just compare them to users who are also in the control for Test #1 so the only variation would be Test #2. – DasBeasto Aug 1 '16 at 17:30
2

Multiple A/B tests not only are very common, but they are extremely useful, way more than simple A/B. However, it has a different name: Multivariate Testing.

Just imagine this: I run a test for page A and page B and decide page B performs better than page A. Great, let's go with page B. Easy, no? Then I decide to test another element, say a button, and button A is better than button B. So I use button A on page B and problem solved.

Now, the question is.... is button A performing better on any instance, or if I test button A on page A it will work better? So... back to point 0, I have to test everything again instead of doing it all at once!

Multivariate to the rescue!

Instead of doing something like the case above, try multivariate testing (or multiple A/B)

Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.

Websites and mobile apps are made of combinations of changeable elements. A multivariate test will change multiple elements, like changing a picture and headline at the same time. Three variations of the image and two variations of the headline are combined to create six versions of the content, which are tested concurrently to find the winning variation.

The total number of variations in a multivariate test will always be:

[# of Variations on Element A] X [# of Variations on Element B] ... =
[Total # of Variations]

enter image description here

Remember, context is everything, so nothing works like in a clean lab. You'll have multiple variables, and the example above is an extremely basic one, let alone if you add features, consider geo, devices, time of the year, audience age and so on.

So, in short: as long as you can define a reasonable subset and test them as much as possible go with multivariate testing

0

There is no need to avoid it, and MVT tests are not a solution, since there is no conceptual difference between A/B and A/B/n tests.

Still, there are some caveats you should be aware of, since there is a theoretical possibility that the tests will influence each other in a way that results in wrong conclusions from at least one of the tests. You might want to read this for a more detailed analysis of the potential issues: http://blog.analytics-toolkit.com/2017/running-multiple-concurrent-ab-tests/

  • If you're testing multiple different versions of several different things, then it's not an A/B test, it is an MVT. An A/B/n test is where you are testing multiple different versions of the same thing. So if you're testing if a blue, black or red button works better. – JonW Aug 4 '17 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.