User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
When I moderate user tests, I sometimes find it hard to get users to verbalise their thought processes - to think aloud.
Users are are very good at give me a running commentary on other things - be that whether they like the graphic design, what they reckon other users might struggle with, how they hate purple, or how they might reword labels. But they do not actually think aloud.
So. How can I explain the think-aloud process in a user test?
Use a rubber duck.
Put a little rubber duck near the user. Tell the user the rubber duck's name - the more non-fitting, the better. Mine is called Frank The Duck.
You see, Frank The Duck is a bit dumb. I tell the user that Frank doesn't know how to use the system, and so I need to teach it. The problem, however, is that since I'm a technical person, I'm really bad in explaining the things that need to be explained in practical, everyday terms - regular English.
So, now, the user has a little humorous mission to undergo with me. I'm putting the user on a position of power, making him teach someone, giving him the reins of the experience. I tell the user that I'm the incompetent one, so the user feel more confident about himself and what should be done. And, them, I incentive the user to explain stuff as he would teach someone.
Teaching is a interesting experience. It puts your brain on a different mode that changes a lot how you think and how you speak, creating some sort of direct bridge between your ideas and your mouth. If the user needs to explain, he will feel that what he needs to do is a bit pointless - you already know what he is doing, why should he bother to do anything like that? However, if the user is going to guide someone into using the system, things change. The user is in control now, he is the guide. He is the responsible for getting the task done.
You can get some experience from Let's Play channels on YouTube. Those people rarely talk when they are playing alone at home. But, when they have an audience - even a silent one - they become really talkative, explaining every little bit of what they are doing, mixing it up with some lateral commentary here and there. Their real audience, however, is just a camera, a lifeless electronic device nearby. Have you ever had a friend by your home while playing videogames, or even a little sibling? The effect is pretty much the same.
Don't ask them to explain - ask them to teach. To guide you. To show you stuff. Let them be responsible for the trip. The extra confidence will give you very nice results.
Jakob Nielsen suggests making and showing participants a 1 minute video of a think-aloud session. In summary, his criteria for such a video are:
He includes an example of such a video in the article above.
I use running questions like:
Steer them away from 'what the parents might have a problem with' - you're not testing the product with their parents or anyone else.
I also start by telling the candidate that the design/construction is not mine so they can't offend me with any negative comments (this is sometimes the source of the "my parents" comments as they want to alert you to a problem but don't want to offend you personally).
I also tell they that we are testing the product and not the user so we need them to behave as normally as possible - if anything doesn't work right, or goes wrong, it's not their fault; it means we have found a bug in the product.
Finally try priming them by telling them the sort of things you want to hear: their thinking about the task they are trying to complete, any problems they are having, any moments of indecision or confusion, and their reasoning for every action.
The purpose of using a think-aloud protocol is to hear what's going on in users' heads as they use (usually) an unfamiliar system. And yeah, it's not an activity that comes naturally.
Similar to @scottishwildcat, I usually demonstrate thinking aloud myself, then giving the subject a short task for them to practice on. (I like that video demo.)
Similar to @Andrew-Martin, I ask prompting questions throughout when subjects go silent. Mostly, "What do you see on the screen?" "What are you looking for?" "What are you trying to do?"
Define you study goals before you come up with questions
Framing the questions right may steer your participants from saying "I like purple"
Try to understand user goals (if possible) beforehand
Sometimes it is helpful to ask participants what their goals are before they start doing a task especially if you are not doing a canned test (pre-defined tasks). It can give you a good insight/understanding why people are clicking/what they are searching for.
Make participants feel comfortable - it's not a test
Participants need to feel comfortable and perceive people who run the "tests" as friendly. Setting the right expectations before the test could help participants to be more open and talk more. The word "test" can scare participants; they may think "oh they are testing me, I might do something wrong". So before starting the tasks, you can explain to participants that you are here to listen and learn from them and it's not a test/evaluation of the person. Also tell them that there is nothing that they can say/do that is wrong
Make them think more about what they do in real life and how it relates to them
"Thinking of the usual things you do with this site, what are you thoughts? Explain some scenarios. What do you find less/more effective for your work/shopping/etc.?"
Give examples of speak aloud
Pretend you are a participant and show them how you can talk
Be a good listener
Ask "why?" questions don't rush into the next task
There were a couple of times when I thought that participants didn't see a link. Asking "why" they didn't click the link explained that they noticed the link but found it irrelevant to their task which is a different problem to solve
Listen to what participants are telling you; sometimes asking new questions and skipping something from the script is more important to learn more
When you hand the participant their first task, say: "Please read this out loud, and then go ahead and do it, and remember to think out loud as you go through the session." If the participant is quiet and has obviously forgotten to think out loud, remind them, by saying: "Please remember to think out loud." Don't ask, "What are you thinking?", or other variations. They may not be thinking anything at all at that point, but before you say anything, be sure they really have forgotten to think out loud.
You don't want to disturb them while they're paying attention to something on the screen, or focusing on a problem. If the participant asks you a question during the study, your first reaction may be to answer it, in order to help them out. However, before you do that, try redirecting them with your own question, like: "What do you think you should do?", or "What would you normally do here?" Once you have their answer, you may choose to provide some information.
It's down to 'managing the session'
If they are focussed on the screen and carrying out the task then I've found that little "Keep Talking" prompts are all they need. You don't want silence on the audio recording.
If they stop looking at the screen and look at you to talk ABOUT the session, then you have to be prepared to 'shut down' that conversation if it starts to not be useful (sometimes it is, sometimes it isn't), and get them back on task looking at the screen.
I agree with the selected questions from Andrew Martin.
Also, be aware that if you're asking the user some kind of justification of his/her acts, then the cognitive load for the overall task is quite heavy (that means that the time taken to perform the task will be affected, and the ability to face new problems might be affected as well; you can't evaluate task efficiency in those conditions). And you will very often have to re-ask the user to explain.
As this kind of think-aloud protocol is quite unnatural, the user needs some time to get used to it. In order to prevent low quality data in the beginning of the session, and to make the user more comfortable, I suggest a pre-session as a training, for at least 15 minutes, on an easy and fun think-aloud task. Then the think aloud will be easier for your user during the main session.
I'm not convinced by a live or video demonstration, which is passive, and which doesn't make the user practice by him/herself. Whereas an easy training session would make the user get used to the think-aloud protocol.
In fact, the point is not to make the user understand what the think-aloud is, but to train him/her for an heavy cognitive load protocol
I am a habitual note–taker. Whenever I am using some new interface or tool, i make so many notes so often that i find it necessary to keep a text editor open in the background at all times.
Often the notes are made for my own design considerations and ruminations, but occasionally i do structure them in the form of comments that could be used to later report bugs or request new features.
Alas for you that not all your testers are interested in u.i. design et similis, but perhaps if you were to tell them that if they see something they don't like or consider to be unintuitive, that they ought to formulate how they would redesign it and to explain why.
On the pros side, this would encourage them to egotize their approach to the interfaces. Maybe their ideas help you expand your own conceptions.
Cons: most users wouldn't know how to correlate their own expectations with functional requirements. They'd diverge off and wouldn't keep it simple, and would laser their focus on any little problem they saw or encountered. Also, you'd need to do some extra processing on the notes if they didn't completely explain the object of their complaint and got honed on the subject of their improvements.
If this is a test where you give the tester full and present attention, then you can always alert them when they are veering too far away from the topic at hand, or otherwise aren't providing useful information.
