What are the pros and cons of getting design recommendations from usability test participants?

Question

I'm looking over a usability report, and one thing is jumping out at me: at the end of each section in which an identified usability issue was discussed, is a section entitled "Recommendations."

Fair enough.

Only it's not recommendations from the UX team that did the tests, it's recommendations they solicited from the usability test participants.

I've not seen this done before and I'm not sure what to think about it, so I thought I'd ask you lot what you think.

What are the pros and cons of getting design recommendations from usability test participants?

Answer 1

I don't think there are any real cons because you can always just ignore the recommendations.

The problem I see is that usually you won't get any new or useful ideas because (with some exceptions of course) users will only know what they want when they see it. Or as Henry Ford put it: "If I had asked people what they wanted, they would have said 'faster horses'."

Answer 2

Cons:

Answer 3

Often the usability test participants will just volunteer comments and a good researcher will dig deeper as to why they like / dislike etc aspects of the site. I'd expect participant comments to be included to illustrate the recommendation from the UX team which did the test rather than being just a list of freestanding comments.

Answer 4

There really isn't a problem with it as long as you just take them as suggestions and not data or fact.

The reason is that users are not designers and thus do not know what the solution to their problem is. Its like asking some college guys how to better a fancy meal. They may know it tastes good or bad but they wont know how to fix it.

Answer 5

In participatory design and co-creation it's common practice to ask participants about their design ideas. Even if the solutions are not as good as what professional designers could think of, truly listening to what people say can lead to really valuable insights in what they want.

On the other hand, theoretically, having people think about design suggestions during a user test could influence their behavior. If you really need the best behavioral data you can get from a controlled experiment, you'd have to be very careful with introducing new factors. But, researchers in for example cognitive psychology have different goals than most people conducting a usability study.

Depending on who reads the report and their understanding of the process, listing the participants' suggestions as 'recommendations' could easily lead to misinterpretations (the UX team doesn't necessarily have to agree with what the participants think).

Answer 6

I always feel like it is toeing a fine line when test participants start giving design advice about the product being tested.

In my experience you can easily get derailed from the tasks at hand because the participant's gaze changes when he/she starts seeing positive feedback about recommendations for the tested product.

I usually try to mark in my notes the areas which prompt suggestions and then go back to them to solicit the advice from the user after the real testing is done. This way the user's thoughts stay on subject while we solve the tasks in the test protocol, but potentially valuable insights aren't lost either.

When interpreting design advice I find that it's important to keep digging for what the root cause is for a user suggestion. Often the user's thought process will originate somewhere foreign to the practitioner and there's a big chance of misinterpreting if we just take the suggestion at face value.

Answer 7

When you ask usability test participants for opinions, comments, or feedback it is no longer a usability test but is now a user interview. In other words it is a focus group consisting of one user.

I think it is ok to ask for opinions at the very end of a test, but only after I have observed the participant try every task that I want to test. In fact this is exactly what we did recently, we ran one task then showed the participant a presentation and asked for opinions.

Answer 8

Because it's their personal opinion. You don't want to satisfy everyone with your design, because you can not. Some people will appreciate the design, some will not.

Maybe all the test participants that didn't sent you design recommendations like the design. For the next time you can even include a question: "What is your general feeling about the design of the website?" so that you get an idea.

Answer 9

Let's remember also that innovation is achieved best(my opinion) through user input in development and design fields. I mean, take a look at Apple, they opened the development of apps to all users, and now we have thousands of apps which are amazing to use, and they were ultimately created by users (approved by Apple developers of course).

So let's not assume that a user giving us recommendations is completely wrong, or does not know what they are talking about. I mean, if someone takes the time to write a recommendation, they most probably have thought about it and come up with some sort of an idea which they thought was legitimate.

So, do users know what they want? I believe that subconsciously they definitely do, so in order for them to realise what they want, they will have to have thought about it, and in this case, if someone has given you a recommendation, we can be sure that they have thought somewhat about it.

My suggestion would be to not get offended or put off if someone does give a recommendation, albeit you don't have to follow what they say, as pointed out before.

-b