What are the best practices for deciding which users to admit to a large scale A/B test on a website? Below are two of my concerns on this choosing process.

Veteran users' aversion to change

Humans are creatures of habit. Once we use something for a long time, we become very efficient and routines take less brain activity. Change, no matter if its for better or for worse, is met with aversion. When we pull the rug from under the user, they become uncomfortable breaking their habit and learning the new process. If we collected stats on veteran user's reaction to a new interface, they would taint the results because we just put a new foreign hurdle in their usual routine.

A personal example of my aversion to change was when I got selected for A/B testing on Google's search engine results page. They had removed the underlines from the result titles. I had sat there staring at the page for many seconds wondering what just happened. The way I parsed the SERP had been engrained in my eyes for 10 years and now it changed. I was shell shocked. I bet you that I, along with others in this A/B test, tainted Google's results with our poor reaction time.

Presumably, the A/B test should be conducted only on new users to avoid human nature's aversion to change. But how do you detect who's old and who's new on a website? A user could have cleared their cookies, so they appear like a new user, but in actuality, could have been using your site for 2 years.

Consistently seeing the same version

Should a particular user only be shown one version of the new interface? For example, once she sees version F, she should continue seeing version F until the A/B testing is done. My theory is that users who are overloaded with too many versions will be confused and taint results.

Technically, how can you provide the same version over and over to the same user? A weak solution is to store a cookie on the user's browser that tells which version they were first randomly assigned to. But cookies don't travel across computers. For instance, I was being served the new Google SERP at home and the old version at work. I was having to adjust to the different versions as I came from work back to home and vise versa. We also can't solve this by tieing a version to an account because users are not gauranteed to be logged in at all times.

1 Answer 1


You should not surprise regular visitors. Show the same variant to repeated visitors. The problem is, as you point out, to know if the user has visited the site before.

  • Using cookies is one solution. Since this is client side, it will not work for the same user from different machines, and the user might have deleted their cookies.
  • Server side logging of IP. You could log the visitors IP and store this information server side. That way you wouldn't be vulnerable to the deletion of the cookies, but same user on different computers will still be a problem. Another issue with this solution is that several users could share the same IP, and these will be provided the same version of the site.
  • Login. If the users can register/login to the site, then this is obviously the best solution to provide the same user the same version, even from different computers.

AFAIK, it is not possible to identify the user across different computers without any form of login function.


You should also take a look at this interesting experiment (and this article).

The point of this site is to identify the browser as uniquely as possible. By taking a lot of factors into account, one can identify a pretty unique "fingerprint" for each browser. If you assume that each user has almost the same setup on different macines, you can pick out some critical factors to determin which variant to show to the visitors. If the same user switches between IE and FireFox, he might expect some variations to the look.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.