1

I conducted an a/b test but I was confused about how to interpret the result. I got 0.5% difference between control and treatment groups and statistical significance for the difference (p <0.05). In my opinion, I can only say that the two groups are different (the change effects treatment group significantly) but how can I interpret the 0.5% difference in observation? Can I say that there is significantly 0.5% difference? When it comes to null hypothesis is "A and B has no difference and I rejected it with p value, I cannot assume how different those two groups are? Then, how should I confirm "~% lift" through A/B test?

Improve this question
1
  • 2
    Given that this sounds like a statistics issue, you might want to post this in the Math StackExchange, or this one as well. But I think Cross Validated is the correct StackExchange to ask this question, they deal with statistics
    – Kitanga Nday
    Jun 25 at 17:16

2 Answers 2

Reset to default
3

You are correct that a significant null hypothesis result only tells you that A and B are different, but it doesn't quite tell you how different they could reasonably be, given the sample size and variability in addition to the observed (sample) difference of 0.5%.

To determine a range on values the population difference could reasonably be, calculate the confidence interval. This will yield a result like, "With 95% certainty, the actual difference between A and B is between 0.28% and 0.79%." The math for confidence intervals is closely related to null hypothesis testing, so if results are statistically significant at the 0.05 level, then the lower bound of the 95% confidence interval should be greater than zero.

Here're a couple ways of calculating confidence intervals with R for the kind of data in A/B testing.

Obviously, of course, you already know the actual difference is most likely about 0.5%. Confidence intervals let you rigorously quantify how big "about" is in your instance.

Improve this answer
0

I am not a statistician, but if you want to more easily see the difference between two values, use the actual difference between your results and not percentage difference.

E.g. "There were 2000 more users doing X than Y" should show a greater difference than "There were 0.05% more users doing X over Y".

Improve this answer
2
  • This is an explanation for how to lie with statistics?
    – Ray Butterworth
    Jun 26 at 0:19
  • @RayButterworth No. But, I assumed this would be how it would be interpreted. It isn't to lie, it's to show the difference. At times 0.05% can be a big deal but you are presenting it to people who don't understand this. So, to avoid all the explaining which takes things nowhere, you present the data in a different format to make it easier for them to see the difference. I remember an article giving percent values that were very close (I think it was 51% vs 50%, but that diff was seen as a big deal). To me, this isn't a problem, to someone who isn't well versed, they won't see it the same way.
    – Kitanga Nday
    Jun 26 at 4:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.