Understanding uplift in A/B testing

Question

Let's say we have a(n) (positive) outcome rate which we try to optimize.

Features affects this outcome.

In a test, we find that users that used a specific feature X have a higher positive outcome rate than users that have not used this feature.

What is then the meaning of uplift regarding to the positive outcome rate? Concretely, how would one measure what the uplift in the positive outcome rate would be if all new users started using feature X?

Example:

• Nb users with positive outcome: 10
• Nb users with positive outcome with feature X: 8
• Nb users with positive outcome without feature X: 2
• Nb new users: 5
• Nb new users with feature X and thus a positive outcome: 3
• Nb new users with feature X without positive outcome: 2

What is then the uplift in the positive outcome if all new users would use feature X?

Answer 1

Optimization tools usually give more statistics than the uplift %. Some will even tell you how many days/weeks the test should run given your current traffic, current conversion rate, and expected uplift.

The two most important things to take note of are the Confidence Level, and Margin of Error.

This usually uses a statistical measurement like Student's T-Test (or similar depending on the software) and can be translated as follows:

Assumptions:

• Version B gets 5% uplift when compared to a Control/Default group.
• There is a 10% margin of error.
• We use a 90% confidence level.

Translation:

In 90% of cases (our chosen confidence level) if you continue to run the experiment for infinite time, Version B (when compared to the control), will get between 4.5% - 5.5% more conversions. Because +/- 10% of a 5% increase is 0.5%.

So it's possible that some times people will convert above or below that range, but 90% of the time we have observed the increase which we can (hopefully) say is due to the change we introduced (our Version B vs. the Control group).

Hope this helps!