I went on the same website on two different browsers, and I got two different layout + design. I was being A/B tested. I didn't like that, probably because:

  1. I don't like to be part of an experiment
  2. I like to have the same experience each time I go to the site

Are there studies / research to measure the effect of A/B testing itself on the user, if the user realizes they are being A/B tested? Will they be less likely to stay on the site?

  • I don't like to be part of an experiment - You're being tested on far more than you realize, so I hope you aren't stressing about it too much. Many companies (Amazon, Google, Faceboook, being among them) will do A/B testing per account - so you always get the same page, but you're getting being compared to other people with different ones. Commented Feb 6, 2015 at 4:51
  • @evil I understand that I'm being ab tested all the time. My concern is that if I use ab testing on my site, my customers might not appreciate it. So just seeing if it should be a concern. And whether there's an alternative
    – dayuloli
    Commented Feb 6, 2015 at 5:02

3 Answers 3


I don't know of any published studies.

I do know from having introduced A/B testing into several organisations that:

  • I've never seen any drop in numbers after we introduced testing.
  • We've never received any complaints.
  • When we've had longitudinal studies when we're revisiting the same users, and they've ended up having slightly different experiences, they've almost always never noticed.

So I'd suspect that the numbers of folk who would both be in a situation to notice differences, and who would consider that are negative, are small enough that you can ignore it.

Of course - this would depend on how you approach your tests. We never split test major UI changes without doing some basic usability testing first to knock off the obvious stupidities.

  • 2
    If I may add - designers shouldn't consider themselves as common users too much. You pay attention to things that casual users don't consciously notice. You take the time to reflect on it etc... Commented Feb 5, 2015 at 10:03
  • Pierre is very correct in saying this. When a designer looks at a site/app, there is a huge discrepancy in the thought process from the user who is not looking at the design but at yet another site. Commented Feb 5, 2015 at 11:01

You are right user behave different when they know they are being tested on. They can get a feeling something is wrong, I know since we sell A/B testing software and early 2012 we got the feeling something was wrong. We were replacing content using jQuery back than after DOM-ready and sometimes people was a blink ... a loading old piece of content that was relatively quickly replaced with the new variation.

We found that when we hardcoded the same variation server-side we could increase conversions by 18%. So we know now from that experience that that installing a heavy Javascript lowers conversion rate, making swaps after DOM-ready could affect conversions up to 18%.

May 2012 we improved the technique and make changes in polling every 50 milliseconds before DOM-ready. Read a bit more on this experience here in the Convert.com blog.

  • I'd suspect foul play, if the site I'm visiting is changing appearence; perhaps the site have been hacked.
    – Lenne
    Commented Mar 23, 2018 at 20:33

I don't know of any studies, but I can give you some advice to help keep your results more meaningful.

If you really want to test the testing software it shouldn't be too hard to write server-side logic or "deployed" vs "not deployed" in some kind of A/A test.

  1. Software Flicker - if you use slow a/b testing platform, you're likely to exhibit page "flicker" which could invalidate tests (i have now seen the default and version B)
  2. Software Latency - if you introduce added latency you're probably harming your conversion rate (ironically). For this reason it's best to dish out the extra money and get a platform that is fast, and backed by SLAs.
  3. What you are measuring - if you test button clicks on the homepage when you're manipulating your purchase funnel, it's likely that you might not see a great deal of change. Your metrics should be as close to the changes as possible... not too upstream, not too downstream.
  4. Test Duration - Don't test for just a few days. The longer you test the closer to the "true average" your numbers will become and converge. You may also notice that as time goes on all versions converge to a similar conversion rate meaning that test didn't do much.
  5. Run A/A tests! - If you run an A/A test for 2 months and you notice uplift, then you should factor that in as an additional margin of error in your real test results. If You see 5% increase, and your next real tests results in a 5% increase, it is possible that it has no little to no effect.

Point #4 is important, personally. At the end of the day, if a visitor wants to purchase from your site they will, regardless of your testing habits. The only exception being things like Price and Benefits offered. But, those (price tests) will probably give you the greatest insights about what drives conversion...not just "do they like blue or green buttons"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.