Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

Short Version

If you run an AB Test, and the results are that there's no difference in your key metric, do you keep the experimental version, or revert the site back to the control?

Story Version

Your client comes dashing into your office.

"I know what'll make this webpage POP! Change the product photo to a buff dude holding a beer, and put the call to action over his chest!"

"Umm...I don't think that really meets the needs of your users, and it doesn't fit with the theme of the..."

"My mother LOVES buff dudes!"

"Umm...ok, how about we A/B Test that, and see if it makes any difference."

So you run the silly A/B Test, and you're somewhat surprised to note that there's no difference. Would it make more sense to just implement the change request, or tell the client that it didn't actually help?

share|improve this question
Obviously a silly story, but you get the point... –  fitzgeraldsteele Apr 13 '12 at 19:48
A silly, but completely believable story. –  DA01 Apr 13 '12 at 20:08

1 Answer 1

up vote 7 down vote accepted

Assuming you're testing a change to an existing system, favor the control. It's what people are used to simply because it's already in place. If tests are inconclusive against users that aren't familiar with your site, at best the change won't help, at worst it briefly confuses current users.

You failed to reject the null hypothesis, stick to what you have.

If it's an A/B test of an unreleased feature...it's a judgement call. Do whatever fits best with the site or standard conventions. Since you didn't notice any difference, it's possible that both fit equally.

However, make sure you've run a proper test; if lots of your tests are inconclusive, it's possible you're testing the wrong changes (it's too little to matter) or you're not testing properly. It's possible that the feature you're A/B testing didn't matter very much, and if that's the case you shouldn't put the extra effort into making the change.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.