What does "Sample size" and "effect size" means in terms of A/B testing? Can anyone explain in simple words with an example?
In A/B testing, effect size is the observed difference in performance between A and B. Take, for example, the following A/B results:
- A: 10 conversions out of 103 visits
- B: 6 conversions out of 97 visits.
So A has a conversion rate of 10/103 = 9.71% while B has a conversion rate of 6/97 = 6.19%. The data suggest that over many visits, A will have 9.71/6.19 – 1 = 57% more conversions. So the effect size, as indicated by relative proportional difference, is 57%. There are other measures of effect size, but relative proportional difference one of the more useful and intuitive ones. You can look at it, and say, “Well, 57% that’s a pretty big difference.” For example, an effect size of 57% implies that you can expect 57% more revenue with A than B, assuming the average value of each conversion is unchanged. That’s a hefty “raise.”
Sample size is the number of visitors in the A/B test, or 103 + 97 = 200 in this example. It’s relevant for the confidence you should have regarding chance effects. While A outperformed B in this test of 200 visits, users were given A or B by a simple digital flip of the coin (if you did it right). Given this element of randomness, it’s possible that A just happened to get more users who would’ve converted whether they got A or B. Maybe there is no real effect of A-versus-B. Maybe A just got lucky.
Intuitively, we sense that the larger the sample size the less likely A will outperform B (or vice versa) just by chance. People use sample size to judge whether they should believe A is actually better than B or dismiss the results as due chance. So, everyone likes to know the sample size.
Sample Size is a Crock
However, the truth is that sample size by itself is almost totally meaningless. You can’t look at a number and make any reliable conclusions on whether A-versus-B has a real effect or not. A chance effect can be reasonably likely with a sample size of 2,000,000. A chance effect can be extremely unlikely with a sample size of 20.
It’s possible (and routine in statistics) to calculate the probability of A and B appearing like they do in the test when A-versus-B has no real effect. That probability is the “p-value.” That’s what you really want to know.
The p-value for the above example is 0.439. If A-versus-B has no real effect, you have a 43.9% chance of seeing results like that. It should be obvious that you shouldn’t get too excited about A’s alleged “superiority.”*
Sample size is a parameter in calculating a p-value, but sample size by itself can be very misleading. For one thing, in typical A/B testing, where conversion rates are very low, a higher number of non-conversions doesn’t matter much one way or the other. For example, consider the following:
- A: 10 conversions out of 1030 visits
- B: 6 conversions out of 970 visits.
The effect size is unchanged –A is still 57% better than B. It’s just that the conversion rates are a tenth of what they were before (probably more realistic too).
But, woohoo! 2000 visitors! Ten times the sample size! But what is the actual p-value? 0.456. Fat lotta good the bigger sample size did.
Effect size affects the p-value. The bigger the difference between A and B in the test, the less likely that chance could produce the result. I mean, sure, A might get a few more ready-to-convert users than B, but a butt-load more? That strains credibility. A can only get so lucky. Consider a sample size of 200, but where A really blows B away:
- A: 18 conversions out of 103 visits
- B: 6 conversions out of 97 visits.
Relative proportional difference is 282% -A’s conversion rate is almost three times B’s. But more significantly (har har*), the p-value is 0.0165. It’s really implausible that A-versus-B has no real effect. I’m convinced that A is really better.
*To interpret p-values, I recommend something close to the scientific tradition of “statistical significance”: a p-value of 0.05 or less should convince you that A-versus-B is a real effect. IMO, p-values around 0.10 should make you suspect there is no real effect, but you shouldn’t necessarily reject the results, especially if the effect size is large. Any result with a p-value of 0.20 or higher shouldn’t be taken seriously regardless of effect size. If there is a large effect size, tell ‘em to keep running the A/B test to see if it holds and the p-value goes down. BTW, I used Fisher’s Exact test to calculate the p-values in this answer, a good choice for an A/B testing.
In (very) simple terms:
Sample size - the number of visitors (participants) included in the A/B test
Effect size - the difference between A and B
A (very) simple example (A/B test results):
- Variation A: 1000 conversions out of 50 000 visitors
- Variation B: 1400 conversions out of 50 000 visitors.
Total A/B test sample size - 100 000 visitors = Variation A + Variation B.
A/B test effect size - Variation B's conversion rate (2.80%) is 40.00% higher than variation A's conversion rate (2.00%).
Of course, there's a lot more to it. Good articles on getting started with A/B testing:
In a brief:
Sample Size: Talks about the number of participants which gives a good result(s) out of the testing. Having too many participants doesn't give you the best results or doesn't help you to find all the defects. Most of the issues will be uncovered by the 5 users. If you involve few more users, they may or may not find new issues, but they are likely to find the same issues. But, the 'number' of users change based on the type of research you are doing.
Effect Size: You can learn more at: http://www.measuringu.com/blog/effect-sizes.php