90

When I moderate user tests, I sometimes find it hard to get users to verbalise their thought processes - to think aloud.

Users are are very good at give me a running commentary on other things - be that whether they like colours, what they reckon other users might struggle with, or how they might reword labels. They will wax lyrical about how they hate purple, or how teal makes an app look 'educational', and generally treat the whole session as an 'opinion gathering' process. But they do not actually think aloud.

So. How can I explain the think-aloud process in a user test?

  • 32
    How can people not like purple? :( – Yates Dec 8 '16 at 12:25
  • 5
    @nocomprende There's a bit of truth in that - if they're not understanding "How was your journey?", then maybe that's not the correct way to be phrasing the question for that audience. (If only there was a discipline which studied how to optimize how people react to things ... ;-) – R.M. Dec 8 '16 at 15:54
  • 5
    Perhaps the issue is that most people are not really used to being aware of what they are doing, and being able to verbalize their process. Most people are simply not introspective in that way. Whatever they do seems so obvious to them that they don't know why they did it, or what any alternatives might be. People who meditate, or study Philosophy or whatever, are more used to seeing their own mind in action. – user67695 Dec 8 '16 at 15:54
  • 3
    It's really simple: I can think, or I can talk. I can't do both at the same time. Observation suggests that I'm far from unique in this. – jamesqf Dec 8 '16 at 18:38
  • 4
    There reaches a certain point in thinking aloud where you're focusing more on what you're thinking aloud and keeping it going than actually using the program. If I were truly thinking aloud, it would be filled with phrases like "Now I'm trying to think to what say next because I haven't formed a coherent idea yet on what to do next and I'm thinking that if I don't come up with something interesting to think about then you'll judge me for it'. – SGR Dec 9 '16 at 9:25
164

Use a rubber duck.

No, seriously!

Put a little rubber duck near the user. Tell the user the rubber duck's name - the more non-fitting, the better. Mine is called Frank The Duck.

You see, Frank The Duck is a bit dumb. I tell the user that Frank doesn't know how to use the system, and so I need to teach it. The problem, however, is that since I'm a technical person, I'm really bad in explaining the things that need to be explained in practical, everyday terms - regular English.

So, now, the user has a little humorous mission to undergo with me. I'm putting the user on a position of power, making him teach someone, giving him the reins of the experience. I tell the user that I'm the incompetent one, so the user feel more confident about himself and what should be done. And, them, I incentive the user to explain stuff as he would teach someone.

Teaching is a interesting experience. It puts your brain on a different mode that changes a lot how you think and how you speak, creating some sort of direct bridge between your ideas and your mouth. If the user needs to explain, he will feel that what he needs to do is a bit pointless - you already know what he is doing, why should he bother to do anything like that? However, if the user is going to guide someone into using the system, things change. The user is in control now, he is the guide. He is the responsible for getting the task done.

You can get some experience from Let's Play channels on YouTube. Those people rarely talk when they are playing alone at home. But, when they have an audience - even a silent one - they become really talkative, explaining every little bit of what they are doing, mixing it up with some lateral commentary here and there. Their real audience, however, is just a camera, a lifeless electronic device nearby. Have you ever had a friend by your home while playing videogames, or even a little sibling? The effect is pretty much the same.

Don't ask them to explain - ask them to teach. To guide you. To show you stuff. Let them be responsible for the trip. The extra confidence will give you very nice results.

  • 11
    I can see that might work in some cases, but personally, I would consider being asked to 'teach' something I'd never seen before to be a bit strange and unhelpful. – calum_b Dec 8 '16 at 14:34
  • 4
    I haven't heard about this method, however it needs to be researched to see its advantages and disadvantages. – Kristiyan Lukanov Dec 8 '16 at 14:58
  • 24
    Related to Rubber duck debugging en.m.wikipedia.org/wiki/Rubber_duck_debugging – spuder Dec 8 '16 at 15:17
  • 12
    @scottishwildcat I think that's where "Frank is a bit dumb" comes in. You can set up a scenario where Frank is trying to use the thing, but realizes he's incompetent - his head is literally full of air, after all. He knows you've never used the thing before, but he's aware that you're better at figuring stuff out than he is (it doesn't take much). So saying "teaching" might be a bit misleading: it's less pedantic lecturing, and more a "let me figure it out with you". Which helps to emphasize that you don't need a polished lecture: verbalizing the stumbling points is okay (and desired). – R.M. Dec 8 '16 at 15:50
  • 5
    If I had to tell stuff to a rubber duck I would think I was being mocked... – poepje Dec 9 '16 at 13:39
23

Jakob Nielsen suggests making and showing participants a 1 minute video of a think-aloud session. In summary, his criteria for such a video are:

  • Use your own staff to act as the participant
  • Don't include picture-in-picture of the participant
  • Show a different UI from the one you are testing
  • Don't focus on any UI issues in the video that you also want to focus on in your real study
  • Don't try to make a 'funny' video

He includes an example of such a video in the article above.

  • Sorry if i've misunderstood but how does this encourage your test candidates to "think out loud"? – Andrew Martin Dec 8 '16 at 14:26
  • 5
    @AndrewMartin It just gives them an example of the sorts of things we'd find useful for them to tell us about while they're using the product. Otherwise, as the OP says, people will sometimes talk about all sorts of stuff that isn't helpful at all, just because they've been told to "think out loud". – calum_b Dec 8 '16 at 14:32
  • 1
    I still don't get it. What you've listed here are a set of requirements for a simple test. The OP didn't ask how to set up a test but specifically how to encourage test candidates to "think out loud". The things you\ve listed may well help the test candidate to focus but won't encourage them to voice their thoughts. – Andrew Martin Dec 8 '16 at 14:35
  • 26
    I'm confused about how you're confused. The video shows participants an example of what you want from them. They see somebody else trying out a UI and thinking out loud. The bullets are suggestions for the video, not suggestions for the subsequent test. – John Kugelman Dec 8 '16 at 15:12
  • Using such a video could add an extra load on the user: "should I say that, will it be relevant or not?". Also, it doesn't train the user to think aloud as there is no practice session, and the user will have difficulties to speak aloud, especially if he's asked to justify his/her actions. In fact, the point is not to make the user understand what the think-aloud is, but to train him/her to lower the impact of this heavy cognitive load protocol. – Yako Dec 20 '16 at 14:16
13

I use running questions like:

  • What are you thinking now?
  • What do you think you need to do next?
  • Why do you want to click that button?
  • What do you expect to happen when you click there?
  • Was that what you expected to happen?

Steer them away from 'what the parents might have a problem with' - you're not testing the product with their parents or anyone else.

I also start by telling the candidate that the design/construction is not mine so they can't offend me with any negative comments (this is sometimes the source of the "my parents" comments as they want to alert you to a problem but don't want to offend you personally).

I also tell they that we are testing the product and not the user so we need them to behave as normally as possible - if anything doesn't work right, or goes wrong, it's not their fault; it means we have found a bug in the product.

Finally try priming them by telling them the sort of things you want to hear: their thinking about the task they are trying to complete, any problems they are having, any moments of indecision or confusion, and their reasoning for every action.

  • 9
    If this doesn't work then I've also found that replaying the full test afterwards helps - showing them each screen they have just seen and then directly asking them 'what were you thinking when you clicked this...'. At that point in the test they've already completed the journey so are able to think more about the process they went through rather than the task they're trying to complete. It's still better to get 'live' feedback, but a post-test review is useful too. – JonW Dec 8 '16 at 11:43
  • 2
    These questions sound a bit creepy, I might want to slap anyone who asked me "why do you want to click that button?" as it seems like an intensely personal way of wording the inquiry. But, I think this is on the right track. – user67695 Dec 8 '16 at 15:58
  • If your next question is "why do you want to slap someone who asks a question like that" you will get slapped for sure. I defer to Iyanla VanZant in matters like this. – user67695 Dec 8 '16 at 16:00
2

The purpose of using a think-aloud protocol is to hear what's going on in users' heads as they use (usually) an unfamiliar system. And yeah, it's not an activity that comes naturally.

Similar to @scottishwildcat, I usually demonstrate thinking aloud myself, then giving the subject a short task for them to practice on. (I like that video demo.)

Similar to @Andrew-Martin, I ask prompting questions throughout when subjects go silent. Mostly, "What do you see on the screen?" "What are you looking for?" "What are you trying to do?"

2

Define you study goals before you come up with questions

Framing the questions right may steer your participants from saying "I like purple"

Try to understand user goals (if possible) beforehand

Sometimes it is helpful to ask participants what their goals are before they start doing a task especially if you are not doing a canned test (pre-defined tasks). It can give you a good insight/understanding why people are clicking/what they are searching for.

Make participants feel comfortable - it's not a test

Participants need to feel comfortable and perceive people who run the "tests" as friendly. Setting the right expectations before the test could help participants to be more open and talk more. The word "test" can scare participants; they may think "oh they are testing me, I might do something wrong". So before starting the tasks, you can explain to participants that you are here to listen and learn from them and it's not a test/evaluation of the person. Also tell them that there is nothing that they can say/do that is wrong

Make them think more about what they do in real life and how it relates to them

"Thinking of the usual things you do with this site, what are you thoughts? Explain some scenarios. What do you find less/more effective for your work/shopping/etc.?"

Give examples of speak aloud

Pretend you are a participant and show them how you can talk

Be a good listener

Ask "why?" questions don't rush into the next task

There were a couple of times when I thought that participants didn't see a link. Asking "why" they didn't click the link explained that they noticed the link but found it irrelevant to their task which is a different problem to solve

Listen to what participants are telling you; sometimes asking new questions and skipping something from the script is more important to learn more

2

When you hand the participant their first task, say: "Please read this out loud, and then go ahead and do it, and remember to think out loud as you go through the session." If the participant is quiet and has obviously forgotten to think out loud, remind them, by saying: "Please remember to think out loud." Don't ask, "What are you thinking?", or other variations. They may not be thinking anything at all at that point, but before you say anything, be sure they really have forgotten to think out loud.

You don't want to disturb them while they're paying attention to something on the screen, or focusing on a problem. If the participant asks you a question during the study, your first reaction may be to answer it, in order to help them out. However, before you do that, try redirecting them with your own question, like: "What do you think you should do?", or "What would you normally do here?" Once you have their answer, you may choose to provide some information.

0

It's down to 'managing the session'

If they are focussed on the screen and carrying out the task then I've found that little "Keep Talking" prompts are all they need. You don't want silence on the audio recording.

If they stop looking at the screen and look at you to talk ABOUT the session, then you have to be prepared to 'shut down' that conversation if it starts to not be useful (sometimes it is, sometimes it isn't), and get them back on task looking at the screen.

  • You could just say "Keep Typing" when they look at you. (see my comment to another answer about getting slapped) – user67695 Dec 9 '16 at 16:06
  • 1
    I usually say, in the preliminary instructions, that I won't answer any questions, then when they look to me for help I'll smile, closed-mouth, and shake my head. – Ken Mohnkern Dec 9 '16 at 21:26
0

I agree with the selected questions from Andrew Martin.

Also, be aware that if you're asking the user some kind of justification of his/her acts, then the cognitive load for the overall task is quite heavy (that means that the time taken to perform the task will be affected, and the ability to face new problems might be affected as well; you can't evaluate task efficiency in those conditions). And you will very often have to re-ask the user to explain.

As this kind of think-aloud protocol is quite unnatural, the user needs some time to get used to it. In order to prevent low quality data in the beginning of the session, and to make the user more comfortable, I suggest a pre-session as a training, for at least 15 minutes, on an easy and fun think-aloud task. Then the think aloud will be easier for your user during the main session.

I'm not convinced by a live or video demonstration, which is passive, and which doesn't make the user practice by him/herself. Whereas an easy training session would make the user get used to the think-aloud protocol.

In fact, the point is not to make the user understand what the think-aloud is, but to train him/her for an heavy cognitive load protocol

0

I am a habitual note–taker. Whenever I am using some new interface or tool, i make so many notes so often that i find it necessary to keep a text editor open in the background at all times.
Often the notes are made for my own design considerations and ruminations, but occasionally i do structure them in the form of comments that could be used to later report bugs or request new features.

Alas for you that not all your testers are interested in u.i. design et similis, but perhaps if you were to tell them that if they see something they don't like or consider to be unintuitive, that they ought to formulate how they would redesign it and to explain why.

On the pros side, this would encourage them to egotize their approach to the interfaces. Maybe their ideas help you expand your own conceptions.
Cons: most users wouldn't know how to correlate their own expectations with functional requirements. They'd diverge off and wouldn't keep it simple, and would laser their focus on any little problem they saw or encountered. Also, you'd need to do some extra processing on the notes if they didn't completely explain the object of their complaint and got honed on the subject of their improvements.

If this is a test where you give the tester full and present attention, then you can always alert them when they are veering too far away from the topic at hand, or otherwise aren't providing useful information.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.