Sign up ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

Short Version

If you run an A/B Test, and the results are that there's no difference in your key metric, do you keep the experimental version, or revert the site back to the control?

Story Version

Your client comes dashing into your office.

"I know what'll make this webpage POP! Change the product photo to a buff dude holding a beer, and put the call to action over his chest!"

"Umm...I don't think that really meets the needs of your users, and it doesn't fit with the theme of the..."

"My mother LOVES buff dudes!"

"Umm...ok, how about we A/B Test that, and see if it makes any difference."

So you run the silly A/B Test, and you're somewhat surprised to note that there's no difference. Would it make more sense to just implement the change request, or tell the client that it didn't actually help?

share|improve this question
Obviously a silly story, but you get the point... –  fitzgeraldsteele Apr 13 '12 at 19:48
A silly, but completely believable story. –  DA01 Apr 13 '12 at 20:08

3 Answers 3

up vote 10 down vote accepted

Assuming you're testing a change to an existing system, favor the control. It's what people are used to simply because it's already in place. If tests are inconclusive against users that aren't familiar with your site, at best the change won't help, at worst it briefly confuses current users.

You failed to reject the null hypothesis, stick to what you have.

If it's an A/B test of an unreleased's a judgement call. Do whatever fits best with the site or standard conventions. Since you didn't notice any difference, it's possible that both fit equally.

However, make sure you've run a proper test; if lots of your tests are inconclusive, it's possible you're testing the wrong changes (it's too little to matter) or you're not testing properly. It's possible that the feature you're A/B testing didn't matter very much, and if that's the case you shouldn't put the extra effort into making the change.

share|improve this answer

I think you have to be careful about the wording here, as a "no change" does not necessarily equate "inconclusive". As an example, an inconclusive result might still show a raise in a metric, but be unusable due to other factors (sampling, underrepresentation, technical issues, etc.). However, not seeing a change, as was mentioned before, should result in favoring the control rather than introducing the change, because this new element might influence other factors your A/B test is not designed to capture and thus you need more testing to see what the change might entail.

share|improve this answer

I believe there are certain cases where applying the changes can be worthwhile. This of course is very subjective to the change. It is ideal to make a guess of the expected result and plan ahead what to do. If a change will give extra benefits that the user won't notice, it can be a good thing to implement. Such as a change that will reduce bandwidth use, reduce charges for the company, etc.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.