5

My question is basically the following:

  • When performing a usability test and utilizing the Think-aloud protocol, is it possible to gather reliable metrics for e.g. time completion rates (i.e. how long did it take to complete the given task)?
  • Is there any user research that exists on how think-aloud protocol potentially affects metrics such as e.g. time completion rate (or similar)?
|improve this question
  • I wonder what would be the point of that on the first place? The two approaches are meant to serve different ends. – drabsv Mar 1 '17 at 15:51
3

Yes is does affect results & yes there is some research ;-)

See, for example Thinking Aloud in the Presence of Interruptions and Time Constraints

Interestingly think aloud may not always make time completion rates longer See The role of instruction and verbalization in improving performance on complex search tasks for example where:

"if people were given both verbal instructions and the requirement to justify each action aloud, performance was improved"

or Effects Of Thinking Aloud Pair Problem Solving On The Troubleshooting Performance Of Undergraduate Agriculture Students In A Power Technology Course where:

Students who participated in TAPPS groups were assigned a listening partner and verbalized their thought processes. They were significantly more successful (p < .05) at troubleshooting engine faults than were students in the work alone control groups. Among students who successfully completed the troubleshooting tasks across both groups,

|improve this answer
  • A footnote to this would be that you can collect some rough and ready completion time data which will point to any large differences in performance. – PhillipW Jun 11 '14 at 20:06
  • Fantastic! Just what I was looking for. :) – Andreas Johansson Jun 13 '14 at 8:01
1

Think aloud protocols are meant to render explicit the user thought process as he/she completes tasks.

This is a clear deviation from how things play-out in normal circumstances as users will be attempting to formalise their thoughts in manner that is intelligible for them and for the tester. This will ultimately affect time taken to complete tasks.

In my opinion, A true and viable account of how users complete a task and how much time they spend doing this needs be based exclusively on observation. This allows users to behave normally and allows the tester to focus on what really counts.

Unfortunately I am not aware of any research done in this area but would suggest to use think aloud protocol when you need to conduct quick usability testing.

Also if you do use think aloud protocol try to detect any spontaneous reactions, frustration, surprise etc and isolate these from other user statements because these could lead to substantial changes to the design or IA. Hope this was of help.

|improve this answer
0

1 - Could you split the testing session into two sections? One focused on the task completion the other more exploratory (think aloud).

Trying to record completion time when you have asked the user to think aloud would potentially add confounding variables into the process. With the typically small sample size of user testing you really would really want to minimise these effects.

2 - I'm not aware of any research relating to this. My recommendation would be separating them if you can. Personally when I have needed quantitative metrics and needed quick results - I presented the users with the two versions of the interface, the task and timed their completion. Using the same user for both versions (within participants design), making sure to alternate the order in which versions are presented to each user.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.