How to rescue a usability test whose participant is lacking confidence?

Question

Last night, I was conducting a usability test with a participant who was very uncertain of his actions. Before the test, I said all of the usual stuff about it being a test of the software and not him, asked him to imagine I wasn't there, that we were hoping he could help us find the software's weaknesses, and so on. The task he was given had also been successfully used with other participants already.

But it still went badly.

The participant became unsure of how to proceed, lost confidence in himself, and he clearly started feeling like it was more of a test of HIM rather than the software. The test dragged on and, despite yielding a good amount of useful feedback, it clearly hadn't been a pleasant experience for him.

How should I rescue such situations in the future? Is there anything I can do to get such sessions back on track, or should I just call an early end to tests where the user gets 'stuck' and stops enjoying it?

I'd welcome any advice...

Answer 1

It's not clear from your question if you have any insight into why the participant lacked confidence. You need to identify this because how you try to solve the problem depends on what's causing the anxiety.

Here are some causes I've found in the past:

• The participant is anxious because of the testing situation. He finds the recording equipment intrusive or he's worried that the test is some kind of elaborate scam.
• The software is very hard to use, making the participant feel stupid.
• The tasks you've set the participant aren't realistic -- he doesn't understand what he's expected to do.
• Something bad is going on in the participant's life (e.g. he had a row with his boss before coming in)
• The participant may have been poorly recruited and doesn't have the expertise necessary to take part.
• The moderator is anxious for some reason and this is being transferred to the participant.

Depending on what's causing it, here's some ideas that might help.

• Start off with an easily achievable task
• Ditch your canned tasks for a self defined task.
• Take a 10-minute break.
• Stop the problematic task and move on.
• Turn off any recording equipment and take written notes instead.
• Stop the usability test and do something else entirely -- for example, turn it into an interview instead.
• Be authentic and talk about the elephant in the room: "You seem quite anxious. Is there anything I can do to help?"

Answer 2

Did you get the tester to use the "think aloud" technique? Sometimes this helps them articulate how they are approaching the task. Maybe they are getting caught up in fine details, or its one specific label that throws them completely off the task at hand. Verbalising as they go is not a natural thing, so bear that in mind as well.

If I strike a similar problem, the user is completely lost, or you can tell they're really not sure, try some prompting.

"I can see you are hovering around the {x} feature, what's drawing your attention to that part of the screen?"

"It looks like your not sure which menu item to choose. Maybe you could take me through what you're thinking. How about the [1st|2nd|xth] menu item, if you clicked that what would you expect to get|where would you expect to go."

Another way I've approached this problem is to take the impact of the problem back. "Wow, it looks like we've really mucked up this design. Would you like to draw me a picture of you think we could do the layout better?"

Answer 3

I used to work in IT for the engineering division, and I've had PLENTY of awkward studies. Personally, I've found that when users lose confidence in their actions, it's because they are just so completely lost and don't want to look like an idiot. It's like when you were in school taking a test you weren't prepared for, and the more you went on, the dumber you felt until it just felt like a completely lost cause until you would just give up. Feeling "dumb" is a painful feeling, and it's your job to minimize that, because if that one users feels that way during testing, then that means some people find the user experience painful too.

What I've found that helps is 4 main things;

1. Reminding them the recordings are completely anonymous

2. Saying that we made it hard on purpose.

   In the study preamble I tell the participant "Sometimes we throw curveballs at you to see how you will handle them, and what kind of workarounds you might come up with. This gives us really valuable feedback, and we don't expect you to know the right answer. We just want to see what you do." This way, when I remind them of this during the test, they don't feel like I'm just placating them.

3. Telling them they're not the only ones who had problems.

   If it's something they seem embarrassed to not know how to do, I will say something like "You know, a lot of other users were stumped here too [even if that's not necessarily true]. Why do you think that is?"

4. Blaming the design

   This kind of goes hand in hand with #3. "Since everybody seems to be stuck on this section, obviously we need to revise this section to make it clearer. How would you expect this to behave?"

5. Reminding them this gives us better data

   If all else fails, I remind users that imperfect tests give us so much more data to work with. If they have problems, others will too, and we want to know what we can do to improve. If a power user blows through the test quickly, it doesn't really tell us what to fix in the application, just that that that particular participant can pick things up quickly.

It's always tough to deal with participants who aren't willing to play along, but most of the time encouragement can go a long way. And sometimes, you just have difficult people, like Phillip said. Sometimes you can adapt to them to get valuable information, but every once in a while, you can't. And that's okay.

Good luck!

Answer 4

I had one of these once.

The guy had decided after about 10 minutes that he thought the software was rubbish - and that he wasn't going to play ball (not couldn't - but wouldn't)

So as David Travis has already suggested I turned it into an interview.

And that produced some very interesting information on what the guy thought the software should have been like.

Answer 5

I know that this question is old, but the original description and its tone are telling. The asker mentions "the usual stuff" as though they're things to check off on a list without really paying heed to them.

The test is said to drag on, but it's not clear why. What did the participant need to do to complete the test? Hopefully the requirements were communicated to him/her beforehand but it's not a given. There should've been "outs" for the participant like a reasonable timeframe before the task ends or the important power to end a test at will.