How to evaluate tasks in a usability test?

Question

I need to do a usability test with 5 participants +- for a system that is like a content mangagement system with crud operations and things like that. So for this I created 5 tasks and now Im with some doubts about a proper way to evaluate them.

Maybe a good way to test the system is verify if the user achieved the goal of the task, then also how hard it was to complete the task, that is the number of necessary actions compared to a otimal number and also the time to achieve the task.

Do you know if this metrics are ok a to do a correct usability test? And if is important to add some more metrics to the test or also if some of the refereed metrics are not very useful for an usability test?

Answer 1

I've always used usability tests for qualitative research. We don't measure anything; we pay attention to where the user does and doesn't have trouble completing the tasks. We have them think aloud and we ask occasional questions.

The report then documents a prioritized list of problems the current system has. We don't compare time to complete or task success rates because usability tests can't control all the variables. (For example, thinking aloud affects completion time.) That's why we can get away with running just 5-7 subjects. We're just looking for problems.

So I guess my advice is to forget metrics and just watch people use the thing.

Answer 2

You are of to a good start. What has worked for me previously is measuring the following during usability test:

• Task success rate Success Rate: The Simplest Usability Metric
• Average time on task, go through all the tasks yourself first to get a baseline on how long it takes a familiar user to complete each tasks so you have something to compare to.10 Essential Usability Metrics
• Task satisfaction 10 Essential Usability Metrics

We recently did a remote usability test on a web app with 25 participants in parallel on a 0$ budget using some slightly experimental set up :-) We published an article on how we approached this Web app remote usability testing on a budget

Good luck & hope you fond this helpful.

Answer 3

You can google for; "Quantitative UX tests" or "Qualitative UX tests" and find some templates. Qualitative are mostly scales 1 to 10, and Qualitative are open questions, satisfaction interviews and so on. Depending on what is your context, choose yourself what you want to know, and what type of information is valuable to you.

Couple of more tips;

- avoid that user is connected to your company anyway, so that he does not have emotional influence prior to test
- make sure that product you are testing is new for user
- tell the user you are not testing him, and that you are testing product, there is no right or wrong
- encourage him to speak loudly his thoughts during session
- give him no hints. if he asks you how to do something, ask back - "what do you think"
- don't go below 4 participants for the tests for results to be statistically correct

Answer 4

User tests are a more qualitative type of evaluation methodology.

Meaning that when you observe the user you find out why exactly things don't work. Not just higher task completion time or other numbers. With a sample size of 5-15 users it will be hard to achieve statistical significance, anyway. So there is no point in overdoing it with stats.

The point of the user test is to learn:

• Where users experience difficulties, and confusions
• How they overcome them without the researcher helping them
• Users mental model or how the user understands how the interface works
• What parts of the interface the user likes, and prefers to use.

Forget about calculating optimal path versus users one. It's just not worth the hassle. You will observe these slowdowns live, and understand much more than time to complete. It is even good to ask questions, about particular problems.

For your measures, you can add number of errors per task, and severity of the errors. It is standard to divide error severity on low, middle, and high, where high is the user cannot complete the task. Just to confirm, time completion, and success rate are good measures to take.

Furthermore, deciding for an appropriate statistical measures depends on the application and context of use. Consequently, you can utilize measures appropriate for your app and context of use.

Finally, make sure to use a think aloud protocol while doing the test, in order to get valuable information of what they think while using the interface. Just ask the participant to think aloud while doing the task.