14

We recently ran an A/B test for a new sidebar in a checkout flow. New variant 70% of traffic, old 30% of traffic. We tried to get client to run at 50/50 but they were sure our version was an improvement, except it delivered a 5% worse conversion rate against the original with 91% significance. 22.5K users (people who interacted with the change) over 7 days.

I'm asking to see if anyone has any literature recommendations or insights on running tests so significantly skewed at this ratio (70/30)?

1
  • 1
    Some good answers below. Basically this 70/30 is an A/B/C/... test. This is easier to understand with 25/75: design 1 is selected for group A (= 25%) and design 2 is selected for group B/C/D (each 25%). The end result will be the average of B/C/D against A. This doesn't make sense and shows how impractical this is. You should go for 50/50 or otherwise you run into calculations as shown in one of the answers. As said in an other answer, you don't have to run the test on 100% of the user base when you want 70% to already experience the new design.
    – jazZRo
    Commented Aug 4, 2023 at 10:34

4 Answers 4

20

Your statistical power is maximized if equal numbers of users are directed to A and B. That is, assuming one design really is better than the other, you’ll achieve statistical significance with the smallest sample size (and fewest number of days of running the test) if there is no “skewing.”

For example, with equal groups of 10,000 each, you might see this:

A B Total
No Conversion 8000 7900 15940
Conversion 2000 2100 4100
Total 10000 10000 20000
Conversion Rate 0.200 0.210
Difference 5.00%
E. Pearson Chi2 3.0678
p 0.0799

Design B outperforms A by 5%, which is statistically significant at the 0.10 level.

However, if we split our 20,000 users 70:30, then, with the conversion rates unchanged, you’d get this:

A B Total
No Conversion 11200 4740 15940
Conversion 2800 1260 4100
Total 14000 6000 20000
Conversion Rate 0.200 0.210
Difference 5.00%
E. Pearson Chi2 2.5958
p 0.1071

A and B are no longer statistically different at the 0.10 level.

It doesn’t matter appreciably which design gets the greater number of users. If B gets 70%, then you get:

A B Total
No Conversion 4800 11060 15860
Conversion 1200 2940 4140
Total 6000 14000 20000
Conversion Rate 0.200 0.210
Difference 5.00%
E. Pearson Chi2 2.5585
p 0.1097

A and B are still not significantly different. There is no statistical advantage to having unequal user groups regardless of which design you think will work better.

In this case, you’d need to increase your sample size to almost 24,000 to match the p-value with equal user groups for A and B.

A B Total
No Conversion 5760 13272 19032
Conversion 1440 3528 4968
Total 7200 16800 24000
Conversion Rate 0.200 0.210
Difference 5.00%
E. Pearson Chi2 3.0702
p 0.0797

If the client is seeking to maximize conversions (e.g., revenue), you could point out that the sooner you get the A-B test over with, the sooner they can reap the rewards of the higher conversion rate of the superior design.

In this case, it’s basically a wash. Running 24,000 users with a 30:70 split resulted in 4968 conversions. But that’s if the client’s bet pays off and B really ends up being superior. If it goes the other way, you get 4872 conversions with a 30:70 split and 24,000 users. If your client has a 70% chance of being right about which design is better (i.e., sensibly equal to the split they want to do), then the expected value is 4939 conversions.

However, running 20,000 users with a 50:50 split results in 4100 conversions. Then the next 4000 user would only get the “winner” (B), which means 0.21 * 4000 = 840 additional conversions, for a total of 4940 conversions out of the same 24,000 users. Hmm. I don’t know if it necessarily always balances out so perfectly (the math is a little complicated to determine that), but it sure doesn’t seem to help to skew the split.

1
  • 1
    This. However, it would be a good idea to ask the client WHY does he think it's superior
    – Devin
    Commented Aug 2, 2023 at 18:59
4

In theory, there isn't a problem with a client-ordered 70/30 test, as long as you're comparing conversion rates and not conversion amounts. If your hypothesis holds up, you'll still see the 5% dip running with 70% of users. It sounds like you have enough of a user base where the conversion rate for the 30% group would still be significant.

Most clients wouldn't want to do this, because they'd lose money while the test runs, but sometimes assurance has its own value. You can simply present the new rate comparison, the new higher significance rate on the 70% group, and the same recommendation. Hopefully the client won't want to lose further revenue by going with the lower-performing variant.

1
  • 1
    Absolutely. But depending on the time lapse, the difference in range may require a regression analysis to actually get the real results
    – Devin
    Commented Aug 2, 2023 at 19:08
4

Run A/B test on 60% of the traffic then. Diverged exactly 50/50 into old and new variants.

Diverge all other 40% of users/sessions/views to new experience (as your client requests), but ignore them in your A/B test, they are not involved in any way.

No one said that A/B test must be ran on all traffic (or that it must be A/B and not A/B/C/D/E, for that matter). I worked in a company with high-volume traffic, so we ran some A/B tests on less than 0.5% of traffic, because that was enough for significant results.

Limiting the traffic throughput for experiment has 2 major benefits (in general, not necessarily in OP's case):

  • You can run other experiments/tests in parallel, on separate traffic.
  • You can have a bit more lax quality requirements for experimental code, as you are going to show it to just a small fraction of users (e.g. if it performs terribly or has some undetected bug, you can stop it before it hits too many users - and you have a lot more time to react than with 50/50 test).

Just take into account how much volume of traffic is now going into experiment - you will need to run it longer to get the same confidence/significance.

2
  • If you ran an A/B test on 1% of your traffic only, that's really a 99.5% / 0.5% split.
    – gnasher729
    Commented Aug 4, 2023 at 12:19
  • 1
    @gnasher729 No, it's 0.5%/0.5% split. The rest of traffic is irrelevant to the experiment. Main point being, it can be used for other experiments without any sort of interference. With sufficiently complex product and many teams you can have a demand for tens or thousands of A/B tests running in parallel, so you need to split the traffic carefully. Of course if you have this capacity, it is almost always better to run big test for shorter, but it's not always possible.
    – Frax
    Commented Aug 4, 2023 at 12:49
2

Your customer wants two things: Find out with evidence which method gives better sales, and don't lose sales during the test. A 50/50 split will give you the evidence quickest. A 70/30 split preferring the better method takes a bit longer to get results, but makes more sales during the test. A 70/30 split preferring the less good method combines the worst things, it takes longer to get the results with evidence, AND you lose sales. Your customer made a bet and they lost.

A good method - if the percentage is just a switch that you can easily change to any value you want - would be to start 70/30 for the method that your customer prefers, and if it looks like their guess was wrong, even with a relative low confidence, then you switch. Running 70/30 for half the test, then 30/70 for the second half, will give you the exact same results as a 50/50 test. Both in confidence and in sales. But if the customer guessed right, then 70/30 for the first half, and staying on 70/30 for the second half, will give you more sales during the test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.