User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
My question is basically the following:
Yes is does affect results & yes there is some research ;-)
See, for example Thinking Aloud in the Presence of Interruptions and Time Constraints
Interestingly think aloud may not always make time completion rates longer See The role of instruction and verbalization in improving performance on complex search tasks for example where:
"if people were given both verbal instructions and the requirement to justify each action aloud, performance was improved"
Students who participated in TAPPS groups were assigned a listening partner and verbalized their thought processes. They were significantly more successful (p < .05) at troubleshooting engine faults than were students in the work alone control groups. Among students who successfully completed the troubleshooting tasks across both groups,
Think aloud protocols are meant to render explicit the user thought process as he/she completes tasks.
This is a clear deviation from how things play-out in normal circumstances as users will be attempting to formalise their thoughts in manner that is intelligible for them and for the tester. This will ultimately affect time taken to complete tasks.
In my opinion, A true and viable account of how users complete a task and how much time they spend doing this needs be based exclusively on observation. This allows users to behave normally and allows the tester to focus on what really counts.
Unfortunately I am not aware of any research done in this area but would suggest to use think aloud protocol when you need to conduct quick usability testing.
Also if you do use think aloud protocol try to detect any spontaneous reactions, frustration, surprise etc and isolate these from other user statements because these could lead to substantial changes to the design or IA. Hope this was of help.
1 - Could you split the testing session into two sections? One focused on the task completion the other more exploratory (think aloud).
Trying to record completion time when you have asked the user to think aloud would potentially add confounding variables into the process. With the typically small sample size of user testing you really would really want to minimise these effects.
2 - I'm not aware of any research relating to this. My recommendation would be separating them if you can. Personally when I have needed quantitative metrics and needed quick results - I presented the users with the two versions of the interface, the task and timed their completion. Using the same user for both versions (within participants design), making sure to alternate the order in which versions are presented to each user.
