Can a page rating with feedback be more effective than a longer usability survey for identifying problem areas?

Question

Our government website has thousands of informational pages, many online self-service task flows and 20+ content editors / stakeholder groups. In the past, periodic usability surveys have been conducted with varied success in determining actionable steps to improve the user experience.

A major challenge is identifying the content that need rework. Content editors are so familiar with the content that it "looks fine to them"... though the customer call center would likely disagree.

We are prototyping and planning to pilot a feedback mechanism at the bottom of each content page similar to the Balsamiq and Apple examples below:

Balsamiq Documentation

OSX Human Interface Guidelines

Our planned solution would gather the quantitative up/down vote and then allow for qualitative free form feedback for a down vote. Data would be ajaxed to the server with minimal interruption to the user's flow.

^{download bmml source – Wireframes created with Balsamiq Mockups}

Foreseen Benefits

• The data could then be segmented as needed by area->application->flow->page
• Usability scores (ratio of +1/-1) could identify problem hot spots
• Qualitative feedback could get content editors started in a targeted manner.

Are we heading in a good direction? Can a page rating with feedback be more effective than a longer usability survey for identifying problem areas?

• Are there any articles or best practices for this type of feedback? It seems distinctly different that the content rating pattern in that it seeks to identify the worst content to pipe back to editors rather than promoting the best content to users. A constant feedback loop like this seems like it would generate valuable data.

Answer 1

Yes, a simple mechanism for feedback can be much more effective than a longer usability survey.

The key for understanding this is to consider the time investment that you're asking your users to make when they give you feedback.

• Almost everyone will be willing to give you a single click of feedback.
• Almost noone will be willing to spend an hour writing an essay - and that one person who does, most likely has a specific axe to grind.

Between the two, the curve shows a pretty fast drop-off.

If you want actionable information, a little bit of data from a whole lot of people will be more useful than detailed information from one or tow.

IIRC, Amazon's own research showed that each additional click in a checkout process causes a significant loss of sales - and this in a process where people wanted to purchase a book. I would expect the dropoff rate for feedback (where the user gets little or nothing out of the action) to be even higher.

I'd suggest you don't need your users to tell you what is wrong, just that they had a problem. The aggregate statistics will pretty quickly identify problem pages, on which you can focus more effort.

Whichever of the suggested approaches you take, make sure you capture every click. Someone selecting "I did not find this article helpful" is useful to know, even if they don't give any reasons, and even if they don't click submit afterwards.

Answer 2

I think its a very good idea what you're doing.

There of course is the alternative simpler 'Yes' 'No' question that Stack Exchange sites put below questions when you're not logged in:

There's a very interesting piece from Jacob Neilsen on a similar topic: Collecting Feedback From Users of an Archive:

1. The Survey Form "ask what topic the user was looking for"
2. Very Small Surveys "just asking a single question"
3. Don't Be Boring; Get Feedback on Live Alternatives "ask an interesting question and give interesting choices"
4. Beyond Simple Surveys "A newsletter asking people for feedback"
5. A "Friends of the Site" User Group
6. User Registration or Profiles

I think 1, 2 and 3 are all quite relevant, so asking just asking the one question is a good idea, focus on asking an interesting question. Perhaps you can improve on the 'was this page helpful' generic question.

Update

I really like the Stack Exchange question at the bottom, not only does it have the natty Star Wars reference, but I think it might be a valid response to ask for in your pages - "what were you actually searching for that brought you to this page". It is easier to answer for the visitors than 'what's wrong with this page'.

Answer 3

You're heading in the right direction. Simple page ratings can be a useful tool in the UI quiver, but keep in mind that other methods, including the usability surveys you mentioned, can help complete the picture.

As others have mentioned your more likely to get negative feedback, but that sounds exactly what you're looking for. Your mock-up is a good example of progressive disclosure -- get the simple Yes/No or Thumbs Up/Down and then give the opportunity to add details.

I happened to notice that Wikipedia seems to be experimenting with this -- I see a simple "Did you find what you're looking for?" rating at the bottom of this page. I also saw that Mediawiki is experimenting with various article feedback tools similar to what we're thinking about here.

Once you do get significant feedback on which pages trigger the most complaints, what do you do? You content creators/editors can try to rethink what's on the page, but this may also be where you'd want to use another usability evaluation tool. Ideally some think aloud testing (which Jakob Nielsen loves) or perhaps remote testing or longer usability surveys.

One final note -- I always close those "We'd like your opinion" popovers that appear when I visit a site (and I work in a field that uses those tools!). I've been thinking that a modified version of that might be useful on sites like you're working on that appears only after extensive browsing by the user. For example, keep a cookie count of how many help pages a user has visited in a session. After hitting a certain threshold, then pop up a link to a survey that says something like "If you're having trouble finding answers, let us know how we can help you". Users who actually take it will likely be those having problems -- they may vent, but you might be more helpful responses to your survey questions.

Answer 4

Your behaviour for actually receiving content complaints seems very neat and sensible to me (although I really don't use these things very often at all).

The question that always arises for me with these controls, though, is why the positive link needs to exist at all.

If, as you describe, the issue is knowing which pages need work, why not just provide a "Report a problem with this article" link and remove the positive links entirely?

The positive link seems to me to be a vestige of a simpler time when we didn't have awesome search engines to find the best pages in a given scope (ostensibly to help categorise "popular" or "useful" pages). You can figure out similar information to what you describe above by creating a ratio of how many negative feedback responses you got into the number of total hits for that page, or simply by using absolute figures of number of complaints to prioritise content updates.

Answer 5

Is quite hard to narrow what users don't like in sites, infact most users don't realize that there is an effort on improve usability, most thinks sites are just content put together.

I suggest systems like UserVoice most of time because users seems to participate, but this system should be used before changes.