I'm curious on the community's opinion about reliability in UX testing.
This is similar to the question: Purpose of Screen Recording but a little broader.
I've discussed with multiple UX designers about their love for A/B testing and other remote tests like screen casts, but coming from a heavily research based background I can't help but feel they're so unreliable. I base this simply on the lack of control of variables.
For example, I've seen designers who test a new landing page with a different call to action (IV) and see an X% improvement of conversion (DV) seem to claim responsibility entirely for this improvement of the DV but there are surely an infinite number of variables in play here that could have affected the DV? Marketing, Pricing, The Users (and the many many variables they come with)?
So I'm keen to know from the community how you analyse and make conclusions from A/B tests & Screen Casts. I've been using the software FullStory (remote screencaster) recently and it's really interesting to see how users are using our stuff but I'm cautious about taking action based on them.
I just can't help but feel if I'd have presented studies during my thesis like these tests my lecturers would have thrown me out!