What are the effects of A/B Testing if the user is aware of it?

Question

I went on the same website on two different browsers, and I got two different layout + design. I was being A/B tested. I didn't like that, probably because:

1. I don't like to be part of an experiment
2. I like to have the same experience each time I go to the site

Are there studies / research to measure the effect of A/B testing itself on the user, if the user realizes they are being A/B tested? Will they be less likely to stay on the site?

Answer 1

I don't know of any published studies.

I do know from having introduced A/B testing into several organisations that:

• I've never seen any drop in numbers after we introduced testing.
• We've never received any complaints.
• When we've had longitudinal studies when we're revisiting the same users, and they've ended up having slightly different experiences, they've almost always never noticed.

So I'd suspect that the numbers of folk who would both be in a situation to notice differences, and who would consider that are negative, are small enough that you can ignore it.

Of course - this would depend on how you approach your tests. We never split test major UI changes without doing some basic usability testing first to knock off the obvious stupidities.

Answer 2

You are right user behave different when they know they are being tested on. They can get a feeling something is wrong, I know since we sell A/B testing software and early 2012 we got the feeling something was wrong. We were replacing content using jQuery back than after DOM-ready and sometimes people was a blink ... a loading old piece of content that was relatively quickly replaced with the new variation.

We found that when we hardcoded the same variation server-side we could increase conversions by 18%. So we know now from that experience that that installing a heavy Javascript lowers conversion rate, making swaps after DOM-ready could affect conversions up to 18%.

May 2012 we improved the technique and make changes in polling every 50 milliseconds before DOM-ready. Read a bit more on this experience here in the Convert.com blog.

Answer 3

I don't know of any studies, but I can give you some advice to help keep your results more meaningful.

If you really want to test the testing software it shouldn't be too hard to write server-side logic or "deployed" vs "not deployed" in some kind of A/A test.

1. Software Flicker - if you use slow a/b testing platform, you're likely to exhibit page "flicker" which could invalidate tests (i have now seen the default and version B)
2. Software Latency - if you introduce added latency you're probably harming your conversion rate (ironically). For this reason it's best to dish out the extra money and get a platform that is fast, and backed by SLAs.
3. What you are measuring - if you test button clicks on the homepage when you're manipulating your purchase funnel, it's likely that you might not see a great deal of change. Your metrics should be as close to the changes as possible... not too upstream, not too downstream.
4. Test Duration - Don't test for just a few days. The longer you test the closer to the "true average" your numbers will become and converge. You may also notice that as time goes on all versions converge to a similar conversion rate meaning that test didn't do much.
5. Run A/A tests! - If you run an A/A test for 2 months and you notice uplift, then you should factor that in as an additional margin of error in your real test results. If You see 5% increase, and your next real tests results in a 5% increase, it is possible that it has no little to no effect.

Point #4 is important, personally. At the end of the day, if a visitor wants to purchase from your site they will, regardless of your testing habits. The only exception being things like Price and Benefits offered. But, those (price tests) will probably give you the greatest insights about what drives conversion...not just "do they like blue or green buttons"