Which version of A/B do I show if you happen to get different versions in two instances?

Question

Let's say I'm doing A/B testing, and this one particular user gets randomly assigned into group B for example. I have a choice right away to either store that on their account in my database, or on their browser. Then tomorrow, that same person visits the website from another device, and randomly gets group A assigned, and sees the A-version of the website. And then, from this new device where they were viewing version A so far, they now sign into their account.

Would it be correct to now load everything B, which has potentially been stored on their account?
Or would it be correct to persist the current visit, and keep showing A in this local browser?
Or, go so far with persisting the new visit to actually store this new version A onto their account for any further page renders from their first device.

This is all assuming that the different versions of the site differ visually, and significantly, so, what the user expects is therefore quite important.

Answer 1

Usually, A/B testing involves comparing the behaviours of users exposed to A with the behaviours of users exposed to B.

Ideally therefore each unique user should be exposed to either A or B, not both. (Unless of course you want to test what happens when a user is exposed to both.)

Answer 2

I prefer to start looking at the problems like, "can users get this website without any medium/device?" The answer is No since they need mediums before it reaches any users. But if the parameter you're trying to measure is independent from the platform, my answer changes.

Besides the reason behind these tests, they should be among the same medium. To say, test A in an iOS environment and test B in an Android device, doesn't necessarily give the exact result with the test results when these tests seperately applied to those platforms.

My answer is, if the problem framed device-oriented before reaching it's targeted users, it should be applyed device based rather than persons. But if you want to measure people anyway, so it would be more accurate to test it platform independent.

• That means a profile might have tons of devices, so you might measure all these plaforms rather than user's profile via insisting different test cases on different devices.

• However, if you're running an A/B test which doesn't necessarily platform-oriented and focusing on measuring user profiles, you'd better measure account based reversely and change the present settings to the user's first touched test medium, in order to preserve accurate statistics.