4

What does "Sample size" and "effect size" means in terms of A/B testing? Can anyone explain in simple words with an example?

Improve this question

3 Answers 3

Reset to default
10

Effect Size

In A/B testing, effect size is the observed difference in performance between A and B. Take, for example, the following A/B results:

  • A: 10 conversions out of 103 visits
  • B: 6 conversions out of 97 visits.

So A has a conversion rate of 10/103 = 9.71% while B has a conversion rate of 6/97 = 6.19%. The data suggest that over many visits, A will have 9.71/6.19 – 1 = 57% more conversions. So the effect size, as indicated by relative proportional difference, is 57%. There are other measures of effect size, but relative proportional difference one of the more useful and intuitive ones. You can look at it, and say, “Well, 57% that’s a pretty big difference.” For example, an effect size of 57% implies that you can expect 57% more revenue with A than B, assuming the average value of each conversion is unchanged. That’s a hefty “raise.”

Sample Size

Sample size is the number of visitors in the A/B test, or 103 + 97 = 200 in this example. It’s relevant for the confidence you should have regarding chance effects. While A outperformed B in this test of 200 visits, users were given A or B by a simple digital flip of the coin (if you did it right). Given this element of randomness, it’s possible that A just happened to get more users who would’ve converted whether they got A or B. Maybe there is no real effect of A-versus-B. Maybe A just got lucky.

Intuitively, we sense that the larger the sample size the less likely A will outperform B (or vice versa) just by chance. People use sample size to judge whether they should believe A is actually better than B or dismiss the results as due chance. So, everyone likes to know the sample size.

Sample Size is a Crock

However, the truth is that sample size by itself is almost totally meaningless. You can’t look at a number and make any reliable conclusions on whether A-versus-B has a real effect or not. A chance effect can be reasonably likely with a sample size of 2,000,000. A chance effect can be extremely unlikely with a sample size of 20.

It’s possible (and routine in statistics) to calculate the probability of A and B appearing like they do in the test when A-versus-B has no real effect. That probability is the “p-value.” That’s what you really want to know.

The p-value for the above example is 0.439. If A-versus-B has no real effect, you have a 43.9% chance of seeing results like that. It should be obvious that you shouldn’t get too excited about A’s alleged “superiority.”*

Sample size is a parameter in calculating a p-value, but sample size by itself can be very misleading. For one thing, in typical A/B testing, where conversion rates are very low, a higher number of non-conversions doesn’t matter much one way or the other. For example, consider the following:

  • A: 10 conversions out of 1030 visits
  • B: 6 conversions out of 970 visits.

The effect size is unchanged –A is still 57% better than B. It’s just that the conversion rates are a tenth of what they were before (probably more realistic too).

But, woohoo! 2000 visitors! Ten times the sample size! But what is the actual p-value? 0.456. Fat lotta good the bigger sample size did.

Effect size affects the p-value. The bigger the difference between A and B in the test, the less likely that chance could produce the result. I mean, sure, A might get a few more ready-to-convert users than B, but a butt-load more? That strains credibility. A can only get so lucky. Consider a sample size of 200, but where A really blows B away:

  • A: 18 conversions out of 103 visits
  • B: 6 conversions out of 97 visits.

Relative proportional difference is 282% -A’s conversion rate is almost three times B’s. But more significantly (har har*), the p-value is 0.0165. It’s really implausible that A-versus-B has no real effect. I’m convinced that A is really better.

Shameless Plug

For more on statistics and usability, see my series of posts. Stat 101 is a non-mathematical overview of the concepts. Stat 203 covers A/B testing.

*To interpret p-values, I recommend something close to the scientific tradition of “statistical significance”: a p-value of 0.05 or less should convince you that A-versus-B is a real effect. IMO, p-values around 0.10 should make you suspect there is no real effect, but you shouldn’t necessarily reject the results, especially if the effect size is large. Any result with a p-value of 0.20 or higher shouldn’t be taken seriously regardless of effect size. If there is a large effect size, tell ‘em to keep running the A/B test to see if it holds and the p-value goes down. BTW, I used Fisher’s Exact test to calculate the p-values in this answer, a good choice for an A/B testing.

Improve this answer
3
  • Sample size does matter however if your sample is very small (such as the often quoted "5" users of qualitative user testing). User testing is rarely done with more than 10-15 participants - so it doesn't produce statistically valid quantative data.
    – PhillipW
    Oct 14, 2016 at 8:33
  • @PhillipW: No, that's a commonly held myth. No matter how small your sample size is, p-value, not sample size, tells you if your results are "statistically valid." Deekshit's link to the nngroup explains why 5 can be statistically valid. For a mathematical explanation, see Lewis, J. R., (2006). Sample sizes for usability tests: Mostly math, not magic. Interactions, 13(6), p29-33. Also see my link to Stat 101 above.
    – Michael Zuschlag
    Oct 14, 2016 at 13:14
  • For you stat geeks, here’s a classroom demonstration where you get statistically significant results with a sample size of 2. Yes, 2. Population: The class. Null hypothesis: At least 90% of the class has 12 fingers. Randomly pick two students and count their fingers. If both don’t have 12, then, using a binomial test, you reject the null hypothesis. Your results, 0 out of 2 don’t have 12 fingers, is statistically significant (p = 0.01). It is “statistically valid” to conclude that less than 90% of your class has 12 fingers.
    – Michael Zuschlag
    Oct 14, 2016 at 13:30
1

In (very) simple terms:

  • Sample size - the number of visitors (participants) included in the A/B test

  • Effect size - the difference between A and B

A (very) simple example (A/B test results):

  • Variation A: 1000 conversions out of 50 000 visitors
  • Variation B: 1400 conversions out of 50 000 visitors.

Total A/B test sample size - 100 000 visitors = Variation A + Variation B.

A/B test effect size - Variation B's conversion rate (2.80%) is 40.00% higher than variation A's conversion rate (2.00%).

Of course, there's a lot more to it. Good articles on getting started with A/B testing:

Improve this answer
0

In a brief:

Sample Size: Talks about the number of participants which gives a good result(s) out of the testing. Having too many participants doesn't give you the best results or doesn't help you to find all the defects. Most of the issues will be uncovered by the 5 users. If you involve few more users, they may or may not find new issues, but they are likely to find the same issues. But, the 'number' of users change based on the type of research you are doing.

You can learn more at: https://www.nngroup.com/articles/how-many-test-users/ http://www.humanfactors.com/newsletters/how_many_test_participants.asp

Effect Size: You can learn more at: http://www.measuringu.com/blog/effect-sizes.php

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.