User Experience Stack Exchange is a question and answer site for user experience researchers and experts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have our own A/B test software which works this way:

  • Every time a user visits a page with a test, he gets a cookie marker.
  • For variant A in test one it may look like this: test1_a
  • For test 2 and variant b: test2_b etc.
  • Once a user makes a purchase, it is written to the A/B panel for the test connected to the marker. For example, if user had the "test1_a" marker, the Variant A in the test 1 will get +1 conversion once a user purchases something.

If we run different tests on a website at once, a user gets multiple markers if he visits different test pages, like:

test1_a, test2_b, etc

After that if user makes a purchase, our A/B software will make +1 sale to EACH test, so

  • +1 sale for varian A in Test 1
  • +1 sale for varian B in Test 2 etc

Is it correct to provide A/B testing using this way?

Note: this is different from providing a multivariative testing on the same page- it is about testing completely different pages of the website at once, taking into account a user may visit all of them.

share|improve this question
So you're showing one user different pages for every visit? – K.. Dec 16 '13 at 8:50
No a user gets same pages fr every visit. – user39061 Dec 16 '13 at 9:02
Are the different pages (and their variants) different parts or steps of a typical task? If so then this is not much different from multivariate testing. Other than that, there's no reason you can't be running multiple independent A/B tests. – Erics Dec 16 '13 at 10:09
a user gets same pages for every visit. What I mean a user may participate in many tests at once, i.e. every page of the site is a single A/B test. – user39061 Dec 16 '13 at 10:27
Whats the difference between the two tests – Mervin Johnsingh Dec 16 '13 at 12:42

There are two problems with your approach:

  1. You're only counting the success - that makes it hard to evaluate performance as your a/b changes - e.g. if you change only b to c, you now have twice as many users who saw a then either b or c. You want to count all views as well as conversions
  2. Maybe it's OK for your scenario, but you're ignoring the user's flow - you can't tell whether the fact that the user saw a1 had an impact when they saw b2.
    In other words, how do you compare the performance of flow a-b-a-a vs. a-b-b-a ?
    It might seem like this is approach lets you test multiple scenarios at once, but I suspect you are confusing your data. You'll get better results from varying a single test at a time - i.e. ensure each user has only a single variant.
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.