20
votes
A/B testing - client wanted the test run 70/30
Your statistical power is maximized if equal numbers of users are directed to A and B. That is, assuming one design really is better than the other, you’ll achieve statistical significance with the ...
- 37.7k
4
votes
A/B testing - client wanted the test run 70/30
Run A/B test on 60% of the traffic then. Diverged exactly 50/50 into old and new variants.
Diverge all other 40% of users/sessions/views to new experience (as your client requests), but ignore them in ...
- 141
4
votes
A/B testing - client wanted the test run 70/30
In theory, there isn't a problem with a client-ordered 70/30 test, as long as you're comparing conversion rates and not conversion amounts. If your hypothesis holds up, you'll still see the 5% dip ...
- 12.1k
3
votes
How can i interpret the observed difference in a/b test?
You are correct that a significant null hypothesis result only tells you that A and B are different, but it doesn't quite tell you how different they could reasonably be, given the sample size and ...
- 37.7k
2
votes
A/B testing - client wanted the test run 70/30
Your customer wants two things: Find out with evidence which method gives better sales, and don't lose sales during the test. A 50/50 split will give you the evidence quickest. A 70/30 split ...
- 476
Only top scored, non community-wiki answers of a minimum length are eligible