A test I've conducted recently reached significance within 24 hours showing a huge drop in conversion. As the site I work on is owned by a large company with high revenue streams, I needed to pause the test to prevent loss of income.

After doing the analysis it turned out that Split C contained a small bug which was enough to pull down all of the results. I started the test again after the fix... same issue happened. 24 hours passed and the same negative effect has been seen.

My task now is to put stoppage guidelines in place to help us decide if a test should be stopped part way through to prevent a detrimental effect to the website - is there any standard measures to use in this case e.g. if CR = -10% then stop the test. Or, is this purely circumstantial?

Any advice around this would be appreciated.

  • 1
    Impossible to rule out experimental design problems with no information about your test. Could be anything (including a valid experimental outcome.) – Luke Smith Nov 14 '17 at 1:50
  • what do you mean by guidelines? Either you stop it or you don't, not sure what is the question here. If you mean guidelines for WHEN, it will depend on many factors, but if there's money involved and you have -10% money on a single day, yes, you have to stop that immediately and try to find out what's going on (and maybe try some testing with users on lab if needed) – Devin Nov 30 '17 at 23:47

For starters, a rigorous Quality Assurance (QA) process is needed before launching a test. Your situation is quite common - find the variant losing only because it contained a bug.

So you need to check if the new variant works as it should on all browsers and devices. Not only that but also if all the goals are set up correctly in your testing tool (often there can be a mismatch - the tool not registering all the conversions/revenue due to incorrect setup). Some checklists you can use:

Second, learn the basic statistics behind A/B testing. This is crucial because data and statistics are tricky and can cause untrained people to see and believe things that are not there (confirmation biases).

Take your situation for example. Your client is freaking out seeing that the new variant is loosing only after 24h. He is convinced that the new variant is costing them money but in reality, it's too early to draw any conclusions. After a few days, the result can be drastically different because the conversion rate and revenue per visitor fluctuate heavily every day (check your analytics you'll likely see a different number every day), meaning that whatever outcome you are seeing after 24h, there's high chance it could be just random.

Some good starting points:

Third, there are basic stopping guidelines you can follow, here's my typical approach (a simplified version):

  • Run the test for at least 7 days, one full week or more precisely one full business cycle (some more expensive products have longer buying cycles - it can take a few weeks for a customer to make a purchasing decision)

After the 7 days:

  • If the variant is loosing, stop the test (and move on to the next test)
  • If the variant is inconclusive, the result has not reached statistically significant difference, stop the test (and move on to the next test)
  • If the variant is a statistically significant winner, run it for another 7 days, just to be sure.

After 14 days:

  • If the variant is still a statistically significant winner, stop the test and implement the variation (and move on to the next test)

But I highly suggested you understand the context (why) before applying them (it's not for everyone), in short:

  • Timing matters - as I mentioned above, your conversion rate fluctuates every day but very likely it is following a trend (e.g conversions are higher on weekends) so the result after 24h is likely just random but after a one full business cycle, it's more believable.

  • Statistical significance is not a stopping rule - by far the biggest mistake businesses make, just because your testing tool says there's a statistically significant difference doesn't mean that there really is a difference. It's just a formula and with the right numbers (like in your case) you can reach it fast but it could be imaginary.

  • One off a/b tests are not that useful - because a/b testing is unpredictable and you'll hit a lot of misses. Therefore with a proper testing program and strategy, you can get more significant wins than misses and thus ensure you actually gain in the long run.

  • You do this to make a business decision, not conduct science - there will be a limit to how precise and accurate your results will be (even if everything seems to be one point) and how much can you learn from each test. Each result is just a prediction after all (not a guarantee). You can certainly improve the accuracy with more sophisticated methods but more often than not this sophistication may not be worth the extra costs (do you want to learn the absolute truths or just make more money?)

Here's some more in-depth information:

But more importantly, everything depends on your A/B testing strategy and your client's business peculiarities and circumstances.

Overall, there really is no one size fits it all solution and guidelines.

You need to know if you actually have the necessary sample sizes or the risk levels your company/client is willing to accept if you have huge volumes of traffic and transactions.

And what is the goal - do you want to learn if certain changes improve your user experience or just want to improve conversion and revenue?

Randomly testing various things (or even best design practices) will not get you very far so you need to have a clear strategy in place.

But don't worry - A/B testing is super hard, even the most experienced pros often create losing variants, it's part of the process:

  • This is an excellent answer, +1. However, OP is mentioning the company is losing money, so he needs to stop immediately. Also, there's not a correct A/B or a pre-existent winner. The idea behind A/B is that you don't know which one is the winner, therefore you have to test and are open to what the results tell you – Devin Jan 8 '18 at 14:55
  • Thanks! If you test two similar versions (A/A testing), after 24 hours, one of the variants (which is 100% identical) will be 'loosing' money and it could be even statistically significant (if you have high volumes of traffic). The point is that after just 24 hours, poorly looking at the numbers you can't yet tell if the company losing money or is it simply a normal fluctuation - there are nuances of course ( e.g. how huge is the drop in revenue?) If the drop is beyond the normal fluctuation, then yeah you should stop immediately and investigate. – Emīls Vēveris Jan 8 '18 at 15:08
  • And the second thing - it all depends on your a/b testing strategy. My approach works for me because my goal is to beat my client's existing page/design - improve their conversion rate and revenue per visitor. I'm not testing random things to see what works best, all changes are backed by research to ensure a high probability of being a winner. With each test, I try to prove or disprove my hypothesis (e.g change "X" will lead to "Y" improvement in revenue because "XYZ"). – Emīls Vēveris Jan 8 '18 at 15:18
  • I guess another factor -- if "A" is "the current design" and "B" is "a new alternative" -- is that if you have a lot of repeat visitors (used to, and expecting "A"), then the first few times they see "B" it will/may slow them down (lower conversion/completion rates) just because it's unfamiliar and they're hunting for what they were expecting. – TripeHound Jan 8 '18 at 15:24
  • 1
    @TripeHound, it is a factor indeed. But you can segment your test results, and compare new vs repeat visitors. Or even leave repeat visitors out of the test. This is something you plan when developing your hypothesis - what are you changing and who are you changing it for (which segment will be included in the test). – Emīls Vēveris Jan 9 '18 at 13:52

We usually stop tests as soon as they achieve statistical significance, which is roughly 5000 clicks for each variant on our platform.

A lot of the A/B test tools use asynchronous tags, which can increase load times. So as soon as we see a clear result, we close the test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.