We recently ran an A/B test for a new sidebar in a checkout flow. New variant 70% of traffic, old 30% of traffic. We tried to get client to run at 50/50 but they were sure our version was an improvement, except it delivered a 5% worse conversion rate against the original with 91% significance. 22.5K users (people who interacted with the change) over 7 days.
I'm asking to see if anyone has any literature recommendations or insights on running tests so significantly skewed at this ratio (70/30)?
Your statistical power is maximized if equal numbers of users are directed to A and B. That is, assuming one design really is better than the other, you’ll achieve statistical significance with the smallest sample size (and fewest number of days of running the test) if there is no “skewing.”
For example, with equal groups of 10,000 each, you might see this:
| A | B | Total | |
|---|---|---|---|
| No Conversion | 8000 | 7900 | 15940 |
| Conversion | 2000 | 2100 | 4100 |
| Total | 10000 | 10000 | 20000 |
| Conversion Rate | 0.200 | 0.210 | |
| Difference | 5.00% | ||
| E. Pearson Chi2 | 3.0678 | ||
| p | 0.0799 |
Design B outperforms A by 5%, which is statistically significant at the 0.10 level.
However, if we split our 20,000 users 70:30, then, with the conversion rates unchanged, you’d get this:
| A | B | Total | |
|---|---|---|---|
| No Conversion | 11200 | 4740 | 15940 |
| Conversion | 2800 | 1260 | 4100 |
| Total | 14000 | 6000 | 20000 |
| Conversion Rate | 0.200 | 0.210 | |
| Difference | 5.00% | ||
| E. Pearson Chi2 | 2.5958 | ||
| p | 0.1071 |
A and B are no longer statistically different at the 0.10 level.
It doesn’t matter appreciably which design gets the greater number of users. If B gets 70%, then you get:
| A | B | Total | |
|---|---|---|---|
| No Conversion | 4800 | 11060 | 15860 |
| Conversion | 1200 | 2940 | 4140 |
| Total | 6000 | 14000 | 20000 |
| Conversion Rate | 0.200 | 0.210 | |
| Difference | 5.00% | ||
| E. Pearson Chi2 | 2.5585 | ||
| p | 0.1097 |
A and B are still not significantly different. There is no statistical advantage to having unequal user groups regardless of which design you think will work better.
In this case, you’d need to increase your sample size to almost 24,000 to match the p-value with equal user groups for A and B.
| A | B | Total | |
|---|---|---|---|
| No Conversion | 5760 | 13272 | 19032 |
| Conversion | 1440 | 3528 | 4968 |
| Total | 7200 | 16800 | 24000 |
| Conversion Rate | 0.200 | 0.210 | |
| Difference | 5.00% | ||
| E. Pearson Chi2 | 3.0702 | ||
| p | 0.0797 |
If the client is seeking to maximize conversions (e.g., revenue), you could point out that the sooner you get the A-B test over with, the sooner they can reap the rewards of the higher conversion rate of the superior design.
In this case, it’s basically a wash. Running 24,000 users with a 30:70 split resulted in 4968 conversions. But that’s if the client’s bet pays off and B really ends up being superior. If it goes the other way, you get 4872 conversions with a 30:70 split and 24,000 users. If your client has a 70% chance of being right about which design is better (i.e., sensibly equal to the split they want to do), then the expected value is 4939 conversions.
However, running 20,000 users with a 50:50 split results in 4100 conversions. Then the next 4000 user would only get the “winner” (B), which means 0.21 * 4000 = 840 additional conversions, for a total of 4940 conversions out of the same 24,000 users. Hmm. I don’t know if it necessarily always balances out so perfectly (the math is a little complicated to determine that), but it sure doesn’t seem to help to skew the split.
In theory, there isn't a problem with a client-ordered 70/30 test, as long as you're comparing conversion rates and not conversion amounts. If your hypothesis holds up, you'll still see the 5% dip running with 70% of users. It sounds like you have enough of a user base where the conversion rate for the 30% group would still be significant.
Most clients wouldn't want to do this, because they'd lose money while the test runs, but sometimes assurance has its own value. You can simply present the new rate comparison, the new higher significance rate on the 70% group, and the same recommendation. Hopefully the client won't want to lose further revenue by going with the lower-performing variant.
Run A/B test on 60% of the traffic then. Diverged exactly 50/50 into old and new variants.
Diverge all other 40% of users/sessions/views to new experience (as your client requests), but ignore them in your A/B test, they are not involved in any way.
No one said that A/B test must be ran on all traffic (or that it must be A/B and not A/B/C/D/E, for that matter). I worked in a company with high-volume traffic, so we ran some A/B tests on less than 0.5% of traffic, because that was enough for significant results.
Limiting the traffic throughput for experiment has 2 major benefits (in general, not necessarily in OP's case):
Just take into account how much volume of traffic is now going into experiment - you will need to run it longer to get the same confidence/significance.
Your customer wants two things: Find out with evidence which method gives better sales, and don't lose sales during the test. A 50/50 split will give you the evidence quickest. A 70/30 split preferring the better method takes a bit longer to get results, but makes more sales during the test. A 70/30 split preferring the less good method combines the worst things, it takes longer to get the results with evidence, AND you lose sales. Your customer made a bet and they lost.
A good method - if the percentage is just a switch that you can easily change to any value you want - would be to start 70/30 for the method that your customer prefers, and if it looks like their guess was wrong, even with a relative low confidence, then you switch. Running 70/30 for half the test, then 30/70 for the second half, will give you the exact same results as a 50/50 test. Both in confidence and in sales. But if the customer guessed right, then 70/30 for the first half, and staying on 70/30 for the second half, will give you more sales during the test.
