We have our own A/B test software which works this way:

  • Every time a user visits a page with a test, he gets a cookie marker.
  • For variant A in test one it may look like this: test1_a
  • For test 2 and variant b: test2_b etc.
  • Once a user makes a purchase, it is written to the A/B panel for the test connected to the marker. For example, if user had the "test1_a" marker, the Variant A in the test 1 will get +1 conversion once a user purchases something.

If we run different tests on a website at once, a user gets multiple markers if he visits different test pages, like:

test1_a, test2_b, etc

After that if user makes a purchase, our A/B software will make +1 sale to EACH test, so

  • +1 sale for varian A in Test 1
  • +1 sale for varian B in Test 2 etc

Is it correct to provide A/B testing using this way?

Note: this is different from providing a multivariative testing on the same page- it is about testing completely different pages of the website at once, taking into account a user may visit all of them.

  • So you're showing one user different pages for every visit?
    – K..
    Dec 16, 2013 at 8:50
  • No a user gets same pages fr every visit.
    – user39061
    Dec 16, 2013 at 9:02
  • Are the different pages (and their variants) different parts or steps of a typical task? If so then this is not much different from multivariate testing. Other than that, there's no reason you can't be running multiple independent A/B tests.
    – Erics
    Dec 16, 2013 at 10:09
  • a user gets same pages for every visit. What I mean a user may participate in many tests at once, i.e. every page of the site is a single A/B test.
    – user39061
    Dec 16, 2013 at 10:27
  • Whats the difference between the two tests
    – Mervin
    Dec 16, 2013 at 12:42

1 Answer 1


There are two problems with your approach:

  1. You're only counting the success - that makes it hard to evaluate performance as your a/b changes - e.g. if you change only b to c, you now have twice as many users who saw a then either b or c. You want to count all views as well as conversions
  2. Maybe it's OK for your scenario, but you're ignoring the user's flow - you can't tell whether the fact that the user saw a1 had an impact when they saw b2.
    In other words, how do you compare the performance of flow a-b-a-a vs. a-b-b-a ?
    It might seem like this is approach lets you test multiple scenarios at once, but I suspect you are confusing your data. You'll get better results from varying a single test at a time - i.e. ensure each user has only a single variant.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.