Reliability of Remote UX Tests (Screen cast, A/B)

Question

I'm curious on the community's opinion about reliability in UX testing.

This is similar to the question: Purpose of Screen Recording but a little broader.

I've discussed with multiple UX designers about their love for A/B testing and other remote tests like screen casts, but coming from a heavily research based background I can't help but feel they're so unreliable. I base this simply on the lack of control of variables.

For example, I've seen designers who test a new landing page with a different call to action (IV) and see an X% improvement of conversion (DV) seem to claim responsibility entirely for this improvement of the DV but there are surely an infinite number of variables in play here that could have affected the DV? Marketing, Pricing, The Users (and the many many variables they come with)?

So I'm keen to know from the community how you analyse and make conclusions from A/B tests & Screen Casts. I've been using the software FullStory (remote screencaster) recently and it's really interesting to see how users are using our stuff but I'm cautious about taking action based on them.

I just can't help but feel if I'd have presented studies during my thesis like these tests my lecturers would have thrown me out!

Answer 1

This is the same as asking 'are surveys reliable?'. The reliability of an experiment depends on the methodology followed, context, participants and time.

There are advantages and disadvantages of lab and field tests. And in both it is impossible to achieve full control of the variables. Yes, in lab tests you have more control but that doesn't mean that this assures reliability.

The strengst of A/B tests:

• produces real objective data
• participants are in their natural context
• no observer effect

Specifically for A/B tests there is a recommendation to run the test for more than 2-3 weeks in order to account for the business seasonality or the variability of the days of the week. For example, the conversion rate in the weekends could be lower compared to weekdays.

Example:

In other words, if you obtain statistical signifince in just 2-3 days I would advise you to run the test for at least two weeks, so that you get more reliable results. This is what you can do to increase the reliability.

Don't worry about what your supervisors will say. You just have to talk with them. They will try to make you base your experiments on well known methodologies so they are safe that the studies will get approved. But this is very limiting as it blocks experiments with novel research methods.

In my opinion, in the UX field the business is moving much faster and generating more UX research methods than in academia.