Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

Our government website has thousands of informational pages, many online self-service task flows and 20+ content editors / stakeholder groups. In the past, periodic usability surveys have been conducted with varied success in determining actionable steps to improve the user experience.

A major challenge is identifying the content that need rework. Content editors are so familiar with the content that it "looks fine to them"... though the customer call center would likely disagree.

We are prototyping and planning to pilot a feedback mechanism at the bottom of each content page similar to the Balsamiq and Apple examples below:

Balsamiq Documentation

Balsamiq Page Rating

OSX Human Interface Guidelines

OSX HIG

Our planned solution would gather the quantitative up/down vote and then allow for qualitative free form feedback for a down vote. Data would be ajaxed to the server with minimal interruption to the user's flow.

mockup

download bmml source – Wireframes created with Balsamiq Mockups

Foreseen Benefits

  • The data could then be segmented as needed by area->application->flow->page
  • Usability scores (ratio of +1/-1) could identify problem hot spots
  • Qualitative feedback could get content editors started in a targeted manner.

Are we heading in a good direction? Can a page rating with feedback be more effective than a longer usability survey for identifying problem areas?

  • Are there any articles or best practices for this type of feedback? It seems distinctly different that the content rating pattern in that it seeks to identify the worst content to pipe back to editors rather than promoting the best content to users. A constant feedback loop like this seems like it would generate valuable data.
share|improve this question
    
What is a "longer usability survey"? Your proposed feedback mechanism is good, but I don't quite understand the alternative you're asking us to compare it to. –  Jonathan Jul 14 '12 at 18:34
    
A 10-15 question survey... More demographic questions, what were you looking for? Did you find it? How hard was it to locate? Did the content meet your needs, etc. –  Luke Charde Jul 14 '12 at 19:31
    
Just bear in mind that this feature must be tested thoroughly (and well designed, of course) to provide you with valuable feedback. It's really annoying to get such questions when they make no sense. Likewise you must remember that this is one kind of feedback. A survey should give you another kind of feedback. (And you are in control of the survey you conduct. With a feedback solution it's all up to the user). –  Jørn E. Angeltveit Jul 20 '12 at 18:34

5 Answers 5

up vote 5 down vote accepted

Yes, a simple mechanism for feedback can be much more effective than a longer usability survey.

The key for understanding this is to consider the time investment that you're asking your users to make when they give you feedback.

  • Almost everyone will be willing to give you a single click of feedback.
  • Almost noone will be willing to spend an hour writing an essay - and that one person who does, most likely has a specific axe to grind.

Between the two, the curve shows a pretty fast drop-off.

If you want actionable information, a little bit of data from a whole lot of people will be more useful than detailed information from one or tow.

IIRC, Amazon's own research showed that each additional click in a checkout process causes a significant loss of sales - and this in a process where people wanted to purchase a book. I would expect the dropoff rate for feedback (where the user gets little or nothing out of the action) to be even higher.

I'd suggest you don't need your users to tell you what is wrong, just that they had a problem. The aggregate statistics will pretty quickly identify problem pages, on which you can focus more effort.

Whichever of the suggested approaches you take, make sure you capture every click. Someone selecting "I did not find this article helpful" is useful to know, even if they don't give any reasons, and even if they don't click submit afterwards.

share|improve this answer
    
Your answer best articulated the rational. Much appreciated. –  Luke Charde Jul 20 '12 at 17:38

Your behaviour for actually receiving content complaints seems very neat and sensible to me (although I really don't use these things very often at all).

The question that always arises for me with these controls, though, is why the positive link needs to exist at all.

If, as you describe, the issue is knowing which pages need work, why not just provide a "Report a problem with this article" link and remove the positive links entirely?

The positive link seems to me to be a vestige of a simpler time when we didn't have awesome search engines to find the best pages in a given scope (ostensibly to help categorise "popular" or "useful" pages). You can figure out similar information to what you describe above by creating a ratio of how many negative feedback responses you got into the number of total hits for that page, or simply by using absolute figures of number of complaints to prioritise content updates.

share|improve this answer
1  
Adding to your idea of not having a positive link, from experience I find people are a lot less prone to give feedback on something they like, but will always voice their opinion if they don't like it, even if it is just a Yes/No. –  Rob Quincey Jul 12 '12 at 11:20
4  
I took the idea from a series of A/B Testing Microsoft did at videolectures.net/kdd07_kohavi_pctce @minute 4:20 where they increased rating participation by 7x by going from a star model rating system to the up down vote. It seems that the commitment for giving an up/down vote is much less than to "report a problem" –  Luke Charde Jul 12 '12 at 12:37
    
The one advantage that the double choice can have is that you also notice contradicting results, that is: a site with many lovers and haters at the same time. This might be interesting information that you wouldn't get with just one negative option. –  Louise Jul 16 '12 at 8:04
    
@Louise: but that presupposes that people who don't leave feedback are neutral about the content. I don't believe that's true: I think people who don't leave any feedback at all may very well have found the content totally satisfactory, but are simply not used to (or unwilling to) "rate" the page when they're done. I believe 50 hits to one page with 1 negative feedback response is probably already contradictory (although I have no evidence to prove it). –  Kit Grose Jul 16 '12 at 9:34

I think its a very good idea what you're doing.

There of course is the alternative simpler 'Yes' 'No' question that Stack Exchange sites put below questions when you're not logged in:

enter image description here

There's a very interesting piece from Jacob Neilsen on a similar topic: Collecting Feedback From Users of an Archive:

  1. The Survey Form "ask what topic the user was looking for"
  2. Very Small Surveys "just asking a single question"
  3. Don't Be Boring; Get Feedback on Live Alternatives "ask an interesting question and give interesting choices"
  4. Beyond Simple Surveys "A newsletter asking people for feedback"
  5. A "Friends of the Site" User Group
  6. User Registration or Profiles

I think 1, 2 and 3 are all quite relevant, so asking just asking the one question is a good idea, focus on asking an interesting question. Perhaps you can improve on the 'was this page helpful' generic question.

Update

I really like the Stack Exchange question at the bottom, not only does it have the natty Star Wars reference, but I think it might be a valid response to ask for in your pages - "what were you actually searching for that brought you to this page". It is easier to answer for the visitors than 'what's wrong with this page'.

enter image description here

share|improve this answer
    
Your Nielsen link allowed me to dig in further into the best practices. Thanks. –  Luke Charde Jul 20 '12 at 17:37

Is quite hard to narrow what users don't like in sites, infact most users don't realize that there is an effort on improve usability, most thinks sites are just content put together.

I suggest systems like UserVoice most of time because users seems to participate, but this system should be used before changes.

share|improve this answer

You're heading in the right direction. Simple page ratings can be a useful tool in the UI quiver, but keep in mind that other methods, including the usability surveys you mentioned, can help complete the picture.

As others have mentioned your more likely to get negative feedback, but that sounds exactly what you're looking for. Your mock-up is a good example of progressive disclosure -- get the simple Yes/No or Thumbs Up/Down and then give the opportunity to add details.

I happened to notice that Wikipedia seems to be experimenting with this -- I see a simple "Did you find what you're looking for?" rating at the bottom of this page. I also saw that Mediawiki is experimenting with various article feedback tools similar to what we're thinking about here.

Once you do get significant feedback on which pages trigger the most complaints, what do you do? You content creators/editors can try to rethink what's on the page, but this may also be where you'd want to use another usability evaluation tool. Ideally some think aloud testing (which Jakob Nielsen loves) or perhaps remote testing or longer usability surveys.

One final note -- I always close those "We'd like your opinion" popovers that appear when I visit a site (and I work in a field that uses those tools!). I've been thinking that a modified version of that might be useful on sites like you're working on that appears only after extensive browsing by the user. For example, keep a cookie count of how many help pages a user has visited in a session. After hitting a certain threshold, then pop up a link to a survey that says something like "If you're having trouble finding answers, let us know how we can help you". Users who actually take it will likely be those having problems -- they may vent, but you might be more helpful responses to your survey questions.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.