Usability Benchmarking and Rating - choosing an approach and rating scale

Question

I’m conducting some UX benchmarking activities and am trying to compare various approaches, both at the individual task level, and activity (multi-task, overview) level. I’m not well versed in the intricacies of the mathematics behind best practices regarding rating scales and would appreciate advice or perspectives on deciding rating scales. I’ll be combining expert reviews with user input too, and want to be intentful on what methods I choose.

My options: The simplest method I’m considering is 3 levels (poor/red/0, average/yellow/1 good, green/2), of course. Of course there is the classic Likert Scale at 5 levels (Strongly negative / 1, negative / 2, neutral / 3, positive / 4, strong positive 5). There are are many other rating systems (thumbs up and down), and 10 point rating systems, to name a few). How might I go about comparing them. I know that various questionnaires have various rating systems, so there must be best practices on choosing one, I’m assuming.

I’m looking for references or explanations on how I might go about thinking about the pros/cons of each - rating scale - I’ll be using different scales at different points, for different reasons, and want to be methodical and have reasoning for my choices. I know this is high level / abstract; I’m not looking for answers per se, but rather I’m seeking a way to help me think this through and criteria / concepts I could use to do so. I’d like to be more informed before I go to my colleagues with some ideas so I can frame the conversation effectively. Thoughts?

Answer 1

My understanding is to first define what type of metrics or KPIs you'd want to track based on which you can decide on the type of scale. I like it how userzoom advocates two types of KPIs which are "Behavioural" & "Attitudinal" and attaches scales accordingly may be you can follow them. In my company we do/call these differently but you get the idea.

Try to understand what are confidence intervals, margin of errors.

An example of a behaviour metric by userzoom is Task Success: Usually represented as a %. Typically, a group of representative users are given a set of realistic tasks with a clear definition of task success.

If 8 out of 10 users completed the task successfully and 2 failed, then ‘Task Success’ would be 80%. Because of the small sample size of 10, the ‘Margin of Error’ at 90% Confidence Level would be about +-25. This means that we are 90% confident that the ‘Task Success’ rate falls somewhere between 55% to 100%.

But if 80 out of 100 users completed the given task successfully, then ‘Task Success’ rate would still be 80%, but with a ‘Margin of Error’ of about 8%. Generally speaking, this means that we are 90% confident that the ‘Task Success’ Rate falls somewhere between 72% to 88%. The larger the sample size, smaller the ‘Margin of Error’.

For Attitudinal

You can use any of the Net Promotor Score (NPS), System Usability Scale (SUS), SUPR-Q (pronounced SuperQ), Customer Satisfaction (CSAT) and more.Look them up to understand the math.

Hope this helps. See more about this here > https://www.userzoom.com/user-experience-research/top-ux-measurements-key-performance-indicators-usability-metrics/

Answer 2

"I’m looking for references or explanations on how I might go about thinking about the pros/cons of each - rating scale - I’ll be using different scales at different points"

You may know this already but it caught me out: Be aware of a wrinkle when analysing your results in that if you add semantic labels to your scales: very good, good, neutral, bad, very bad etc, a lot of people then regard that data as being ordinal rather than interval/continuous even if you also have a number scale attached (1-5), the argument being that the difference between say very good and good, and good and neutral, are not the same. Thus, different analysis methods are needed:

http://blog.minitab.com/blog/adventures-in-statistics-2/best-way-to-analyze-likert-item-data%3A-two-sample-t-test-versus-mann-whitney

Answer 3

SUS for overall system usability

For general usability research, the System Usability Scale works well. It has a long history so you can benchmark against expected norms, even by generalized product types. I use an "all positive" variation, but the resulting scores are the same.

Event tracking for the details

Asking what users think is fun and occasionally useful, but nothing is more telling than real-world behavior. Event patterns like feature engagement and task success ratios observed in real day-to-day scenarios is gold.

When you design a feature, whether you're prototyping or in production, there should always be a target for success. Ask yourself this important question:

What usability metrics relate to my feature success metrics?

When a project is considered "landed" and successful, don't forget about those metrics. You'll build up a catalog of health indicators that can be implemented in a dashboard for on-going product monitoring.

This might be a controversial position here …
If a usability issue doesn't impact your success metrics it may not matter.