4

Let's say we have a(n) (positive) outcome rate which we try to optimize.

Features affects this outcome.

In a test, we find that users that used a specific feature X have a higher positive outcome rate than users that have not used this feature.

What is then the meaning of uplift regarding to the positive outcome rate? Concretely, how would one measure what the uplift in the positive outcome rate would be if all new users started using feature X?

Example:

  • Nb users with positive outcome: 10
  • Nb users with positive outcome with feature X: 8
  • Nb users with positive outcome without feature X: 2
  • Nb new users: 5
  • Nb new users with feature X and thus a positive outcome: 3
  • Nb new users with feature X without positive outcome: 2

What is then the uplift in the positive outcome if all new users would use feature X?

13
  • I think you need to make this a little clearer - your title says: Understanding Uplift in A/B Testing, yet the example makes no reference to A or B. – SteveD Aug 12 '16 at 9:16
  • Basically what I meant was that a feature X has been implemented and now we want to check whether that feature positively enhances the positive outcome rate. I.e. is there an uplift when users use this feature? That's the A/B testing part. – JohnAndrews Aug 12 '16 at 10:22
  • Your example is a little confusing:-) Nb = Number? I don't understand why you are mentioning new users here. – SteveD Aug 12 '16 at 10:34
  • Number indeed. Well I am looking into understanding what the impact would be if all New users would use feature X. In the current data there were 5 new users and of them 3 had implemented a feature X and thus a positive outcome. Now, if we were to make a forecast, what could this data tell us? – JohnAndrews Aug 12 '16 at 10:41
  • The whole point of an A/B test is to measure current vs new - it is not a forecasting measure. Obviously you can make a prediction (scientific or not) and then see if the prediction comes true in the A/B. This is why I was confused by the question. – SteveD Aug 12 '16 at 10:44
2

Optimization tools usually give more statistics than the uplift %. Some will even tell you how many days/weeks the test should run given your current traffic, current conversion rate, and expected uplift.

The two most important things to take note of are the Confidence Level, and Margin of Error.

This usually uses a statistical measurement like Student's T-Test (or similar depending on the software) and can be translated as follows:

Assumptions:

  • Version B gets 5% uplift when compared to a Control/Default group.
  • There is a 10% margin of error.
  • We use a 90% confidence level.

Translation:

In 90% of cases (our chosen confidence level) if you continue to run the experiment for infinite time, Version B (when compared to the control), will get between 4.5% - 5.5% more conversions. Because +/- 10% of a 5% increase is 0.5%.

So it's possible that some times people will convert above or below that range, but 90% of the time we have observed the increase which we can (hopefully) say is due to the change we introduced (our Version B vs. the Control group).

Hope this helps!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.