Should I ask the user to complete the task as fast as she can in user test?

Question

I need to evaluate insurance auto quote process which has 3-4 pages of forms to fill. I have designed a prototype with two variations and want to test which of them is easier to use. So I plan to make a standard user test with think aloud protocol and test 10 users on 1 variation and 10 on the other variation.

The question is should I ask them to complete the task as fast as they could or no, in order to get the most valid and valuable data.

I need to quantify the data so I can present some stats, like task completion time to the managers.

I'm thinking that if I instruct them to fill it as fast as they could they might omit to share some valuable insights.

In your experience what are the pros and cons?

Thanks in advance.

Edit:

The problem is that there is no actual system which I can track statistics on it and do A/B test. Thats not available. I need to elicit abstract findigs for the form filling process because we are building a insurance software which will be implemented in different countries where different rules apply. So I need abstract findings for the insurance form filling process, as a whole.

Answer 1

I'd say no. You're asking them to voice the reasoning behind their actions as they go along (think-aloud protocol) - they may decide one way to speed things up is not to say much which would be counter-productive.

I'd also be wary about trying to mix qualitative observation techniques with quantitative measurements. Slower users may just be observing more and telling you more in a think-aloud test. If the managers want numbers for a chart collect data on the number of mistakes made and/or the number of observations collected. Save speed measurements for another test.

Answer 2

No, you shouldn't ask a user to complete the task as fast as possible.

A user test is too much of an artificial setting to get a true read of efficiency. It's also hard for you to get enough testers to get statistically significant data for comparison.

Time to completion is better analyzed by tracking metrics on the actual system. You should set up an A/B tests for your system for the 2 layouts.

Answer 3

I feel one should never ever instruct the user,let him go with his flow and in that process you as UX designer should observe him,each user could be a an important persona for you. Based on the research,work on your form to reduce the pain points that you observed while taking down sessions with the users.

Answer 4

As other people mentioned, it's best not to ask users to go through the tasks as fast as they can. There are a few reasons, one is what you already mentioned, the more the users share with you while talking aloud the more time they take to complete the tasks. Going through a usability test and talking aloud are not natural for the participants and require more cognitive resources compared to when people perform tasks in their own context. They might take longer and make more mistakes while talking to you in a lab setting.

If your goal is to evaluate the usability and understand which design is more intuitive, the think aloud protocol helps you with understanding the whys. Usability and ease of use goes way beyond time on task, but it is possible that your managers think of them synonymously. On the other hand, if your goal is to get quantitative data maybe an A/B test is a better approach, in that case 10 people is not statistically significant.

I would separate the quantitative and qualitative studies, but if you need to get both types of data in one study, maybe starting with a passionate task where participants go through the task as they would normally do and then follow up with think-aloud will not be a bad idea. For the passionate task instead of asking them to go through it as fast as they could, I would recommend asking them to imagine themselves in a scenario/situation in which they have to act fast.

Answer 5

As others have said, you shouldn't try influence the user in any certain direction for a usability test. Take a look at this test script, I found it really helpful in terms of framing the usability test and instructing the users to think out loud etc.

In terms of analytics/data from your usability test, you could try using Canvas Flip - I haven't used it before but looks interesting!

Answer 6

You need to spend a lot of time planning and thinking about what you want to get out of the test. When performing a usability tests you should be looking to gather quantitative and/or qualitative data.

As a rule of thumb, in terms of time, a typical usability test can be broken down into three phases:

• Planning - 45% of time

• Executing the test - 10% of time

• Evaluating the results/produce reports - 45% of time

So if you decide you want to evaluate time, because it is important that users can quickly reach their goal due to some business need, then it is acceptable to ask them to complete the task as quickly as possible. You will be noting speed and accuracy. However, this is just one aspect of your usability test. You might add other things you are looking for, e.g. how do they feel about the task they just completed, and you may notice other things that happen during the test, especially if you are asking people to speak aloud about what they are thinking.

This is why the planning phase is very important.

Answer 7

If you talk about restricting the time, please take a look at benchmark usability testing.

If you want to give valuable insights, let the user talk and let them explain what they see, what they are doing.,

If I still want user to complete the task "as fast as possible", I'd give them scenarios like this: you have 1 minute to catch the subway, while you're going down the escalators, please try to "complete my task" and catch the subway.