What do you do with an inconclusive A/B Test? Keep the change or lose it?

Question

Short Version

If you run an A/B Test, and the results are that there's no difference in your key metric, do you keep the experimental version, or revert the site back to the control?

Story Version

Your client comes dashing into your office.

"I know what'll make this webpage POP! Change the product photo to a buff dude holding a beer, and put the call to action over his chest!"

"Umm...I don't think that really meets the needs of your users, and it doesn't fit with the theme of the..."

"My mother LOVES buff dudes!"

"Umm...ok, how about we A/B Test that, and see if it makes any difference."

So you run the silly A/B Test, and you're somewhat surprised to note that there's no difference. Would it make more sense to just implement the change request, or tell the client that it didn't actually help?

Answer 1

Assuming you're testing a change to an existing system, favor the control. It's what people are used to simply because it's already in place. If tests are inconclusive against users that aren't familiar with your site, at best the change won't help, at worst it briefly confuses current users.

You failed to reject the null hypothesis, stick to what you have.

If it's an A/B test of an unreleased feature...it's a judgement call. Do whatever fits best with the site or standard conventions. Since you didn't notice any difference, it's possible that both fit equally.

However, make sure you've run a proper test; if lots of your tests are inconclusive, it's possible you're testing the wrong changes (it's too little to matter) or you're not testing properly. It's possible that the feature you're A/B testing didn't matter very much, and if that's the case you shouldn't put the extra effort into making the change.

Answer 2

I think you have to be careful about the wording here, as a "no change" does not necessarily equate "inconclusive". As an example, an inconclusive result might still show a raise in a metric, but be unusable due to other factors (sampling, underrepresentation, technical issues, etc.). However, not seeing a change, as was mentioned before, should result in favoring the control rather than introducing the change, because this new element might influence other factors your A/B test is not designed to capture and thus you need more testing to see what the change might entail.

Answer 3

I believe there are certain cases where applying the changes can be worthwhile. This of course is very subjective to the change. It is ideal to make a guess of the expected result and plan ahead what to do. If a change will give extra benefits that the user won't notice, it can be a good thing to implement. Such as a change that will reduce bandwidth use, reduce charges for the company, etc.