Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I'm looking over a usability report, and one thing is jumping out at me: at the end of each section in which an identified usability issue was discussed, is a section entitled "Recommendations."

Fair enough.

Only it's not recommendations from the UX team that did the tests, it's recommendations they solicited from the usability test participants.

I've not seen this done before and I'm not sure what to think about it, so I thought I'd ask you lot what you think.

What are the pros and cons of getting design recommendations from usability test participants?

share|improve this question
add comment

9 Answers

up vote 12 down vote accepted

I don't think there are any real cons because you can always just ignore the recommendations.

The problem I see is that usually you won't get any new or useful ideas because (with some exceptions of course) users will only know what they want when they see it. Or as Henry Ford put it: "If I had asked people what they wanted, they would have said 'faster horses'."

share|improve this answer
2  
+1 for one of my favorite design quotes of all time –  jonshariat May 11 '11 at 20:52
3  
:D thanks - in return here's my other often used quote: "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." - Antoine de Saint-Exupéry ;) –  Phil May 11 '11 at 21:11
6  
Another good one! Here is one of the ones I quote to my managers every so often... "You can use the eraser on the drafting table or a sledge hammer on the construction site. - Frank Lloyd Wright –  jonshariat May 11 '11 at 21:17
    
Brilliant, never heard it - I have a feeling I'm gonna use this one a lot :) Thanks! –  Phil May 11 '11 at 21:25
    
the con is you may be setting user expectations that their suggestions will be implemented –  jk. May 12 '11 at 10:47
show 3 more comments

There really isn't a problem with it as long as you just take them as suggestions and not data or fact.

The reason is that users are not designers and thus do not know what the solution to their problem is. Its like asking some college guys how to better a fancy meal. They may know it tastes good or bad but they wont know how to fix it.

share|improve this answer
add comment

Often the usability test participants will just volunteer comments and a good researcher will dig deeper as to why they like / dislike etc aspects of the site. I'd expect participant comments to be included to illustrate the recommendation from the UX team which did the test rather than being just a list of freestanding comments.

share|improve this answer
    
Nice thought about using it to illustrate decisions. That was, in essence, where I ended up with this. The report is written as "they said it so we'll do it" which I'm opposed to. –  gef05 May 12 '11 at 13:12
add comment

In participatory design and co-creation it's common practice to ask participants about their design ideas. Even if the solutions are not as good as what professional designers could think of, truly listening to what people say can lead to really valuable insights in what they want.

On the other hand, theoretically, having people think about design suggestions during a user test could influence their behavior. If you really need the best behavioral data you can get from a controlled experiment, you'd have to be very careful with introducing new factors. But, researchers in for example cognitive psychology have different goals than most people conducting a usability study.

Depending on who reads the report and their understanding of the process, listing the participants' suggestions as 'recommendations' could easily lead to misinterpretations (the UX team doesn't necessarily have to agree with what the participants think).

share|improve this answer
1  
Yes, I think the distraction and the setting of expectations is an issue. The users were asked at the end of each task for suggestions - which means that after the first task was complete, they then knew that for all remaining tasks they were going to be asked for suggestions. Not an appropriate mindset. –  gef05 May 12 '11 at 13:19
add comment

Let's remember also that innovation is achieved best(my opinion) through user input in development and design fields. I mean, take a look at Apple, they opened the development of apps to all users, and now we have thousands of apps which are amazing to use, and they were ultimately created by users (approved by Apple developers of course).

So let's not assume that a user giving us recommendations is completely wrong, or does not know what they are talking about. I mean, if someone takes the time to write a recommendation, they most probably have thought about it and come up with some sort of an idea which they thought was legitimate.

So, do users know what they want? I believe that subconsciously they definitely do, so in order for them to realise what they want, they will have to have thought about it, and in this case, if someone has given you a recommendation, we can be sure that they have thought somewhat about it.

My suggestion would be to not get offended or put off if someone does give a recommendation, albeit you don't have to follow what they say, as pointed out before.

-b

share|improve this answer
2  
But note that said app designers had absolutely no input in development of the platform and hardware itself. Steve Jobs is well known for not soliciting focus group type feedback (which is a good thing, in Apple's case) –  DA01 May 12 '11 at 2:12
    
Yes, I must agree with DA01 on this. Jobs has a reputation for not looking to focus groups. –  gef05 May 12 '11 at 13:16
    
I also agree. Thanks @DA01. –  BorhanMooz May 12 '11 at 23:45
add comment

I always feel like it is toeing a fine line when test participants start giving design advice about the product being tested.

In my experience you can easily get derailed from the tasks at hand because the participant's gaze changes when he/she starts seeing positive feedback about recommendations for the tested product.

I usually try to mark in my notes the areas which prompt suggestions and then go back to them to solicit the advice from the user after the real testing is done. This way the user's thoughts stay on subject while we solve the tasks in the test protocol, but potentially valuable insights aren't lost either.

When interpreting design advice I find that it's important to keep digging for what the root cause is for a user suggestion. Often the user's thought process will originate somewhere foreign to the practitioner and there's a big chance of misinterpreting if we just take the suggestion at face value.

share|improve this answer
add comment

When you ask usability test participants for opinions, comments, or feedback it is no longer a usability test but is now a user interview. In other words it is a focus group consisting of one user.

I think it is ok to ask for opinions at the very end of a test, but only after I have observed the participant try every task that I want to test. In fact this is exactly what we did recently, we ran one task then showed the participant a presentation and asked for opinions.

share|improve this answer
add comment

Cons:

enter image description here

share|improve this answer
    
:D +1 for the most funny answer ever! –  Phil May 12 '11 at 12:14
    
Oh yes. Homer's vehicle came to mind while I was reading the reports, don't you worry about that. –  gef05 May 12 '11 at 13:12
    
Why doesn't this have hundreds of upvotes is totally beyond me :) –  Mahn Jul 11 '12 at 17:06
add comment

Because it's their personal opinion. You don't want to satisfy everyone with your design, because you can not. Some people will appreciate the design, some will not.

Maybe all the test participants that didn't sent you design recommendations like the design. For the next time you can even include a question: "What is your general feeling about the design of the website?" so that you get an idea.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.