Skip to main content
20 votes

A/B testing - client wanted the test run 70/30

Your statistical power is maximized if equal numbers of users are directed to A and B. That is, assuming one design really is better than the other, you’ll achieve statistical significance with the ...
Michael Zuschlag's user avatar
4 votes

A/B testing - client wanted the test run 70/30

Run A/B test on 60% of the traffic then. Diverged exactly 50/50 into old and new variants. Diverge all other 40% of users/sessions/views to new experience (as your client requests), but ignore them in ...
Frax's user avatar
  • 141
4 votes

A/B testing - client wanted the test run 70/30

In theory, there isn't a problem with a client-ordered 70/30 test, as long as you're comparing conversion rates and not conversion amounts. If your hypothesis holds up, you'll still see the 5% dip ...
Izquierdo's user avatar
  • 12.7k
3 votes

How can i interpret the observed difference in a/b test?

You are correct that a significant null hypothesis result only tells you that A and B are different, but it doesn't quite tell you how different they could reasonably be, given the sample size and ...
Michael Zuschlag's user avatar
2 votes

A/B testing - client wanted the test run 70/30

Your customer wants two things: Find out with evidence which method gives better sales, and don't lose sales during the test. A 50/50 split will give you the evidence quickest. A 70/30 split ...
gnasher729's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible