Guidelines for stopping an A/B (or MVT) test early due to negative metrics

Question

A test I've conducted recently reached significance within 24 hours showing a huge drop in conversion. As the site I work on is owned by a large company with high revenue streams, I needed to pause the test to prevent loss of income.

After doing the analysis it turned out that Split C contained a small bug which was enough to pull down all of the results. I started the test again after the fix... same issue happened. 24 hours passed and the same negative effect has been seen.

My task now is to put stoppage guidelines in place to help us decide if a test should be stopped part way through to prevent a detrimental effect to the website - is there any standard measures to use in this case e.g. if CR = -10% then stop the test. Or, is this purely circumstantial?

Any advice around this would be appreciated.

Answer 1

For starters, a rigorous Quality Assurance (QA) process is needed before launching a test. Your situation is quite common - find the variant losing only because it contained a bug.

So you need to check if the new variant works as it should on all browsers and devices. Not only that but also if all the goals are set up correctly in your testing tool (often there can be a mismatch - the tool not registering all the conversions/revenue due to incorrect setup). Some checklists you can use:

• http://www.digital-tonic.co.uk/digital-tonic-blog/how-to-run-qa-quality-assuranceon-your-ab-tests/
• https://www.prwd.co.uk/blog/the-qa-process-you-should-follow-before-every-ab-test/

Second, learn the basic statistics behind A/B testing. This is crucial because data and statistics are tricky and can cause untrained people to see and believe things that are not there (confirmation biases).

Take your situation for example. Your client is freaking out seeing that the new variant is loosing only after 24h. He is convinced that the new variant is costing them money but in reality, it's too early to draw any conclusions. After a few days, the result can be drastically different because the conversion rate and revenue per visitor fluctuate heavily every day (check your analytics you'll likely see a different number every day), meaning that whatever outcome you are seeing after 24h, there's high chance it could be just random.

Some good starting points:

• Thinking about A/B Testing for Your Client? Read This First.
• A/B Testing Statistics: An Intuitive Guide For Non-Mathematicians
• Ignorant No More: Crash Course on A/B Testing Statistics

Third, there are basic stopping guidelines you can follow, here's my typical approach (a simplified version):

• Run the test for at least 7 days, one full week or more precisely one full business cycle (some more expensive products have longer buying cycles - it can take a few weeks for a customer to make a purchasing decision)

After the 7 days:

• If the variant is loosing, stop the test (and move on to the next test)
• If the variant is inconclusive, the result has not reached statistically significant difference, stop the test (and move on to the next test)
• If the variant is a statistically significant winner, run it for another 7 days, just to be sure.

After 14 days:

• If the variant is still a statistically significant winner, stop the test and implement the variation (and move on to the next test)

But I highly suggested you understand the context (why) before applying them (it's not for everyone), in short:

• Timing matters - as I mentioned above, your conversion rate fluctuates every day but very likely it is following a trend (e.g conversions are higher on weekends) so the result after 24h is likely just random but after a one full business cycle, it's more believable.

• Statistical significance is not a stopping rule - by far the biggest mistake businesses make, just because your testing tool says there's a statistically significant difference doesn't mean that there really is a difference. It's just a formula and with the right numbers (like in your case) you can reach it fast but it could be imaginary.

• One off a/b tests are not that useful - because a/b testing is unpredictable and you'll hit a lot of misses. Therefore with a proper testing program and strategy, you can get more significant wins than misses and thus ensure you actually gain in the long run.

• You do this to make a business decision, not conduct science - there will be a limit to how precise and accurate your results will be (even if everything seems to be one point) and how much can you learn from each test. Each result is just a prediction after all (not a guarantee). You can certainly improve the accuracy with more sophisticated methods but more often than not this sophistication may not be worth the extra costs (do you want to learn the absolute truths or just make more money?)

Here's some more in-depth information:

• Statistical Significance Does Not Equal Validity (or Why You Get Imaginary Lifts)
• Stopping A/B Tests: How Many Conversions Do I Need?
• How to Minimize A/B Test Validity Threats
• My 7 Rules for A/B Testing That Triple Conversion Rates

But more importantly, everything depends on your A/B testing strategy and your client's business peculiarities and circumstances.

Overall, there really is no one size fits it all solution and guidelines.

You need to know if you actually have the necessary sample sizes or the risk levels your company/client is willing to accept if you have huge volumes of traffic and transactions.

And what is the goal - do you want to learn if certain changes improve your user experience or just want to improve conversion and revenue?

Randomly testing various things (or even best design practices) will not get you very far so you need to have a clear strategy in place.

But don't worry - A/B testing is super hard, even the most experienced pros often create losing variants, it's part of the process:

• How to Build a Strong A/B Testing Plan That Gets Results
• How to Choose the Right A/B Testing Strategy for Your Clients

Answer 2

We usually stop tests as soon as they achieve statistical significance, which is roughly 5000 clicks for each variant on our platform.

A lot of the A/B test tools use asynchronous tags, which can increase load times. So as soon as we see a clear result, we close the test.