What are the best practices for deciding which users to admit to a large scale A/B test on a website? Below are two of my concerns on this choosing process.
Veteran users' aversion to change
Humans are creatures of habit. Once we use something for a long time, we become very efficient and routines take less brain activity. Change, no matter if its for better or for worse, is met with aversion. When we pull the rug from under the user, they become uncomfortable breaking their habit and learning the new process. If we collected stats on veteran user's reaction to a new interface, they would taint the results because we just put a new foreign hurdle in their usual routine.
A personal example of my aversion to change was when I got selected for A/B testing on Google's search engine results page. They had removed the underlines from the result titles. I had sat there staring at the page for many seconds wondering what just happened. The way I parsed the SERP had been engrained in my eyes for 10 years and now it changed. I was shell shocked. I bet you that I, along with others in this A/B test, tainted Google's results with our poor reaction time.
Presumably, the A/B test should be conducted only on new users to avoid human nature's aversion to change. But how do you detect who's old and who's new on a website? A user could have cleared their cookies, so they appear like a new user, but in actuality, could have been using your site for 2 years.
Consistently seeing the same version
Should a particular user only be shown one version of the new interface? For example, once she sees version F, she should continue seeing version F until the A/B testing is done. My theory is that users who are overloaded with too many versions will be confused and taint results.
Technically, how can you provide the same version over and over to the same user? A weak solution is to store a cookie on the user's browser that tells which version they were first randomly assigned to. But cookies don't travel across computers. For instance, I was being served the new Google SERP at home and the old version at work. I was having to adjust to the different versions as I came from work back to home and vise versa. We also can't solve this by tieing a version to an account because users are not gauranteed to be logged in at all times.
