3

The company I work for (a deal roundup site) has redesigned the look of our deal information tiles. (Changing the layout of how the item's image, price, and other information is arranged.) We're going to A/B test to see if the new design is more engaging to our users... however, there's some debate as to whether or not we should message our users about this change, before they see it. The anti-messaging people fear that priming an audience before such a change will yield tainted test data. Everything else I've read online so far says it's dangerous not to prime an audience before a UI change.

So, my question is this: Does messaging a UI change before it's implemented sully the results of an A/B test in any way? Are there any pros to pulling a surprise UI change on users?

Thanks, in advance, for any help or advice. I'm a newly-minted UX Researcher and I'm glad to have found your community, as I'll need all the help I can get!

1

I think you're confusing the methodology. A/B test are ran at the same time, so if you simply replace a site with a completely new version of the site, it doesn't matter if you let people know or not, the samples will provide other kind of results.

While in research, you'll always want an "as clean as possible, not contaminated sample", so this alone would ask your question, the important part is that you wouldn't be making an A/B test, so no A/B rules should be considered.

Just for reference, and because it will help you if you're new to A/B, take a read to A Beginner’s Guide To A/B Testing: An Introduction and specifically this part:

Tests need to be run simultaneously to account for any variations in timing. You can’t test one variation today and the other one tomorrow, because you can’t factor in any variables that might have changed between today and tomorrow. Instead, you need to split the traffic seeing your variations at the same time.

Also, you might have noticed here and there how big sites show some tiny variations randomly: if you see that, they're conducting A/B tests. And mentioning this because of a very important thing you'll need to consider: A/B is for small changes, not complete redesigns

  • Ah, yes, I see where I may have used incorrect terminology. The UI change is not being implemented, but just tested. It should be a true A/B where a part of our audience sees the deal tiles on our homepage in the new design, the other will not. So, then, if this is a true a/b test, we would NOT message it. However, if we were just going to go all-in and roll this new tile to all users, messaging would be very important. – Jeff Somogyi Aug 2 '16 at 21:48
  • 1
    Ah, OK. In that case, it will depend. Most sites don't do it, but if your user base is used to some flow, and this change may affect this flow, then yes, it's recommended that you let them know about these changes and implement some kind of tour using walkthroughs or coach marks – Devin Aug 2 '16 at 23:05
  • Thank you for your input. Since this is an A/B test, but of a large/important part of our site's flow, I think I will recommend that we message our test group about it (as they've been rather vocally against site changes in the past). Thanks again! – Jeff Somogyi Aug 3 '16 at 15:10
  • I wouldn't say that A/B tests are for small changes only. It's to get numbers for changes you aren't sure about (and you can never be sure how users will react to certain changes, unless it involves basic stuff like changing from a dropdown with 2 choices to radio buttons or the likes). If you want to see what kind of effect a new feature has on a platform then you can absolutely use A/B testing to compare key metrics. – Daniel Slowacek Aug 4 '16 at 7:36
  • @DanielSlowacek, while what you say is technically true, in real life doing split testing over a complete redesign includes so many different variables your results will always be unreliable. Literally, you'll be comparing apples and oranges. In these cases, you use KPI analysis tools rather than A/B – Devin Aug 4 '16 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.