58

Recently I was lucky enough to sit right next to a user on the train who happened to be using one of the apps that I designed. Without disclosing this to them, I managed to observe how they use the app. Even though they weren't thinking out loud as in a user testing sessions it was very insightful to see how they use it.

Which comes to the question, compared to bringing in users for a 'user testing' sessions, would their behavior be different when users know they are being observed? Or even worse, when an incentive is given. If so would that make the research flawed?

Improve this question
4
  • 43
    So, how did it go? Were they using the app like you expected? Just curious.
    – Erik Bennett
    Commented Jul 17, 2018 at 16:03
  • 5
    We started to simulate the "real world scenarios" and stopped observe them directly. We tested an indoor navigation system for a train station. Instead of see how they use it, we just recorded the smartphone screen and gave them some tasks to do. Then the test users were off on their own for about 30mins. In the stress situation of a busy train station, they forgot pretty fast that it is just a test. Some of them started swearing or just clicked everywhere on the screen when something didn't worked. If possible, I only test like that. Put them in a stress situation, and they act naturally.
    – Michael Schmidt
    Commented Jul 18, 2018 at 7:50
  • 4
    "If possible, I only test like that. Put them in a stress situation, and they act naturally." -- That's a good plot for a reality show.
    – Diogo Kollross
    Commented Jul 18, 2018 at 13:36
  • @ErikBennett yes but not exactly. they were taking their time and scrolling back and forth the catalog, which is haven't seen in user tests. This could be because they were genuinely looking to buy something online when using the app.
    – Blue Ocean
    Commented Jul 19, 2018 at 3:14

3 Answers 3

Reset to default
70

Yes, it's known as the Hawthorne Effect and it's one of many cognitive bias' to be aware of when running usability testing sessions.

From Wikipedia

The Hawthorne effect (also referred to as the observer effect) is a type of reactivity in which individuals modify an aspect of their behavior in response to their awareness of being observed.

Does it make the research flawed? Well yes, in a way. But that doesn't mean it's not worth doing. You can mitigate it to some extent - making the testing facilities as natural and comfortable as you can, not standing there in a lab-coat with a clipboard, generally being friendly and human and not like a machine. And (importantly) telling the user up-front that it's not them that you're testing, it's the system.

But bias will always be there, you need to be aware of it, try testing with multiple methods and users and don't assume after the testing that how people behaved during the test is exactly how they'll behave in 'the real world'.

But you still get useful test data from people, even if they know they're being monitored. People will try even harder to solve problems in a lab than they would at home, so if they can't figure out how to get to the checkout on your eCommerce store when you're testing them in a lab then you really know you've got a problem - because people at home would've bailed out long ago.

Improve this answer
5
  • 2
    Ahh takes me right back to my psychology undergraduate days :-)
    – Lisa Tweedie
    Commented Jul 17, 2018 at 10:58
  • 4
    Even bringing users in has an effect. They'll use the app differently in your lab than they do in their own kitchen or on the bus.
    – Ken Mohnkern
    Commented Jul 17, 2018 at 13:51
  • The ultimate answer is probably that there's not much you can do about it. It's hard to do widespread testing in the wild, so you do the best you can in the lab.
    – Barmar
    Commented Jul 17, 2018 at 16:17
  • 11
    This effect is why, buried in a great many EULAs, you'll find something about usage data -- in short, the company remotely tracks what you're doing, while you use the app, without you having to know about it. This normally isn't a big issue since they're also controlling the servers and could get the information another way, but it can be extremely helpful to figure out that, say, no one knows a menu shortcut for an otherwise tedious process exists.
    – anon
    Commented Jul 17, 2018 at 16:18
  • It also depends on what the app does, example: games are best tested by gamers in a beta test environment. The app designer also has to wear many hats.
    – Joe R.
    Commented Jul 18, 2018 at 3:11
5

As the answer from JonW states there is the Hawthorne Effect.

But there are some behaviours you may be able to see in a natural environment if you take the ethnographic research approach rather than a user testing one.

User testing is more commonly done in an artificial environment set up by the user testers, some with varying levels of a natural feel to the user. Ethnographic research is done purely in the field, in the users own environment. It may raise some other issues that user testing may not pick up on.

For example if you are observing someone at their desk, do they have post-its around with passwords, guides etc on how to do things, this is an indication that the UX needs improvements to make things easier. User testing may not pick up on that. So while you where observing the user use your app on the train, it may have presented some other things that wouldn't have happened in a lab.

I wouldn't say its flawed, I'd say it provides a different perspective and some alternative results and observations that user testing in a lab environment may not highlight and vice versa.

Improve this answer
1

As the other answers point out, there's a name for this phenomenon, and it's flawed.

But even observing the user without them knowing is flawed too, simply because you don't know what they're thinking. They might hate a certain feature but put up with it because they have no choice.

The most flawless testing/observation that you can get is to use your own software. I don't mean just use it as part of your job of testing the software but really have the need to use such a software just like your intended audience does.

That's actually how many high-profile software products are created, e.g. Basecamp - they probably use Basecamp to manage the development of Basecamp. Gmail - people in Google probably use Gmail to communicate with each other. React - people in Facebook probably use React to make their website.

On the other hand, a Netflix movie will have random glitches if you try to watch it with closed captions in Chrome. That's probably because no one in Netflix has ever used Chrome to watch a Netflix movie with closed captions.

Improve this answer
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.