Drawbacks to user feedback

Question

User feedback is the information provided by users after using your product or site, usually provided via form, email or in person. It can be solicited or unsolicited, though my question focuses on solicited feedback. Obviously, feedback should not be your sole source of information in determining UX design. Otherwise, you could just throw up a feedback button on every site and remove the need for UX professionals altogether.

What are the pros and cons of user-provided feedback versus other UX information-gathering methods(testing, observation, etc.)?

Answer 1

There is a saying "don't listen to your users, watch them", perhaps first suggested by Jacob Nielsen here.

To design an easy-to-use interface, pay attention to what users do, not what they say. Self-reported claims are unreliable, as are user speculations about future behavior.

If you search for "don't listen to your users" you'll find other recommendations along this line of thinking.

Answer 2

There are many ways of collecting user feedback. Asking for opinions about the product is one way to get feedback, which is often called "self-reported usability feedback" (or "self-reported usability survey").

I understood that your question is related to that kind of feedback, which is not particularly bad as long as you analyze the results in context and combine them with other methods.

The things to watch out in a "self-reported usability feedback" are:

• Open ended questions are hard to analyze. If you get feedback in person, interviewing people can give you a lot of insight. But if you do it in a electronic form with lots of people they are not so useful.

• The effectiveness of a survey that uses Likert scales, depends a lot on how you present the question and scale. Don't take the survey design lightly.

• Give the users some privacy or anonymity to answer questions. For example if you give a feedback form in person, and you are there waiting for the response, you'll get feedback biased towards positive responses.

• When you analyze survey results which contain scales (ie Bad...Excellent), don't do a direct average because there is no defined distance between points of the scale and a simple average makes no statistical sense. The usual approach is to group the responses in two areas (ie: Bad/Good) and count the responses that fall into those areas.

• You'll not get design ideas from this kind of feedback. That's what the quote "don't listen to your users, watch them" means. The trick is in what "watch them" actually means. A better way to "watch" users is by doing activities. That approach is often called "Participatory Design", and consists mainly in organizing activities with users so you can get more insight ("Participatory" in that context doesn't mean that the people take the design decisions). You can find some example activities here: http://www.frogdesign.com/collective-action-toolkit

Ok, I've mentioned a lot of bad things to watch out, is there anything good? Yes:

• It's a cheap way of getting lots of feedback. So you can analyze some trends: if most of the people hate some part of the app, it's clear that you need to re-think it
• You can use it as a trigger for other kinds of usability studies

Answer 3

Pros:

• some comments may trigger thoughts that hadn't been previously considered

Cons:

• people don't always know what they want
• the users are often lacking big-picture context
• they are giving feedback in the context of how they understand the technology works, rather than what the technology can potentially do

None of that infers that you should not collect user feedback, but rather just always accept it for what it is...additional info that may or may not help the overall process.

Answer 4

Your vision of feedback only as of information gathering method is too narrow. I like both @obelia and @DA01 answers in context of information gathering.

But thinking wider, feedback from the users' point of view is:

• Communication tool. Users use feedback as fast and easy way to communicate with company, no matter of feedback content.
• Control tool. Users use feedback as the control tool. They believe their feedback can influence some company decisions or/and warn other users.
• Assessment tool. Watching others' feedbacks users infer possible troubles or/and benefits of making deal with the company.

Every point itself has value for users and creates good UX. Of course, company uses feedback as information gathering method, too, considering its limitations.

So, thinking of feedback, consider user needs first, then business goals.

Answer 5

I'd also consider user's motives for bothering to leave feedback. If user's don't feel that anything will happen as a result of their feedback, then they probably won't bother taking the time to leave feedback in the first place.

Also consider that one reason for leaving feedback is the

"I so hated your product / service that it will make me feel better to have a rant at your organisation ! "

kind of feedback. Which may skew the results.

Answer 6

Some great answers here, but there is one point not mentioned: non-representativeness of the user sample when self-selected.

Put simply, if feedback is optional, content users do not leave feedback. Delighted users might sometimes leave feedback, if they feel altruistic. Angry users love to give feedback, to demand that somebody makes a change. The result is that you only get to know the opinion of a vocal minority, and it is hard to gauge whether the silent majority agrees with them or not. In the worst case, you may start making changes for the complainers and thus break features most users loved but did not bother to praise.

This does not mean that user feedback is bad per se, but that you must factor in this possibility before making decisions. For example, you can let the users provide an optional opinion for months or years, gather some change candidates, then make a survey (possibly paying the participating users) where the users have to commit to taking it before they have seen the questions. Once you have such a pseudo-random selection, your representativeness should be good enough for decision making based on the resulting data.

Answer 7

You can also mention heuristic reviews- they are equally important and can provide great results. I never liked getting too many feedback and then dealing with what to change or what not- simply get expert feedback (may be 2-3 experts if possible) and then implement it. Usability testing is too confusing and not effective at all (my opinion)- we use notableapp.com for exchanging feedback within team if needed and FeedbackGuru.com for getting expert feedback.