When evaluating prototypes, how can I ensure the user evaluates specific aspects instead of the product/design as a whole?

Question

I've heard about this issue from different people and written sources. When building prototypes one would like evaluation from users. One type of evaluation is, for each functionality, whether it is actually needed or appreciated and how it should be constructed.

What I hear is that if a complete, or rather complete, prototype is presented, the user will comment on irrelevant things, and not be able to give useful feedback on the functionalities. Not even when instructed to do focus on specific aspects. For instance the user will comment on the graphical aesthetics, maybe even comment on colours and font face.

So what I hear being recommended is that in some way I should present prototypes that focus solely on the aspects I want to evaluate. If there are five functionalities that I need tested, I could make five groups of prototypes. The first group could focus on a the presentation of historical data, fx a plot with some buttons for zooming and sliding. I could then make three different versions of this.

I then see two basic ways of making the user focus on this functionality, when evaluating it.

1. Graying out every other aspect of the GUI.
2. Simply not showing the rest of the GUI. But there must be other ways as well.

What is the best way to achieve this?

Answer 1

User performance not user opinions

The solution is to do a proper usability test. Don’t show users a prototype and ask them what they think of it. That’s asking them to imagine what it would be like to use the product, which yields unreliable data. Instead, have them use the prototype so they (and you by your measures) know what it’s like to use the product. To get the users and the data focused on a specific functionality, have users use the prototype for a task you select that should use that functionality.

Is the functionality needed or appreciated?

If they don’t use the functionality, then maybe the users believe it’s unnecessary. Or maybe they didn’t realize it’s there, which is a separate problem to address. Interviewing the users after the test will settle which it is. If the users don’t use the functionality, point it out to them, and instruct them re-do the task with the functionality. Once they’re done, re-interview them, asking them to compare doing the task with and without the functionality. Combine that subjective data with objective data on how easily and how well the task was actually completed (allowing for the fact that any task is easier the second time through). Keep in mind that users cannot easily separate design from inherent usefulness. Some users may dismiss a functionality they tried not because it’s a bad idea, but because it was too hard to use as designed.

How should the functionality be designed?

In any case, once you have users using the functionality, now you can observe how well the UI works for the users and thus how it should be constructed. You see what in fact is and isn’t a problem, rather than hypothetically what the user thinks would and would not be a problem. An interview afterward will help you narrow down the possible causes of any problems.

Use a limited fidelity prototype?

All that said, a “prototype” isn’t necessarily a full mock-up of the product. You can develop only an aspect of the product you want to test. For example, if you just want to compare your design of a pie menu versus an ordinary context menu, you just have to mock up the two menu designs. It doesn’t even have to be an actual set of menu items for the product you’re ultimately designing. However, the purpose of making such low fidelity mockups is more to save you work (and money) by not developing a lot of stuff that may (will) need to be changed based on the usability test results. It’s not necessarily for focusing the user. If you already have a high-fidelity prototype anyway, use it.

Answer 2

A/B testing. Create two prototypes one which has the features, you are looking to test, the other one does not. If the storyline is the same and the user is able to complete the tasks this product is designed to do then by default those features are useless. You can also create prototypes that only focus specifically on the features that need testing and not the entire user flow.