Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

So we have a site that processes the booking of a travel tour over 4 or 5 pages in a wizard format.

A while ago I put a feedback section on the final page of that process. This consisted of a rating of your "booking experience" on a scale of 1-10 as well as a free-text comments section.

Now our site has pretty much been UX-designed by developers (it has been graphic-designed by a designer so looks pretty) which means it's not as usable as it could be, and we get a good few of the basics wrong. I expected average to above-average scores from the 1-10 rating.

Over 20 thousand ratings later and a huge proportion of them are 9 and 10 out of 10. A lot of people seem to think it's flawless!

Now I'm aware that the average user wouldn't know what improvements to make even if they could say that a particular page/section was not very easy-to-use, so maybe they have that thought and when they can't think of anything that is obviously improvable they decide to vote a 9 or 10.

But I'm also aware that people tend to blame themselves when they can't get the product to do what they want. Consequently, if they are at all frustrated during the booking process and then do finally make it to a page that has a nice big "Booking Confirmed" on it, they are instantly overloaded with utter euphoria and the endorphin-rush is ruining the objectivity I need for accurate feedback data.

How do I control for the users' positive state of mind ("Yay I've just made a booking!") when asking for a feedback rating?

share|improve this question
1  
xkcd.com/937 –  Erics Oct 19 '11 at 6:38
    
You are saying that you have put the feedback section on the final page of the process. All the users who have a problem during the process are unhappy users but you don't know it because they don't even reach the feedback section. –  Nicolas Barbulesco Dec 19 '13 at 15:23
add comment

5 Answers 5

up vote 12 down vote accepted

Maybe the feedback rating is not what you need in this case.

It sounds like you're interested in discovering potential problems with the process. But those who actually complete the wizard aren't having problems.

So instead, what if you create links to appropriate help topics off to the side of every page? Those who are having problems will tend to click through the help. And then through whatever analytics system you're using, you can then review trends where people are having problems and which specific help topics they're looking at. As time goes on, you can use these insights to improve both the help and the wizard itself.

share|improve this answer
1  
Like e.commerce apps, though, putting distracting elements on every page of the process may only serve to distract. Hard to say if asking for feedback this way will inadvertently affect booking process or completion rates, as well. –  jameswanless Feb 14 '11 at 17:17
    
I think the idea should be to create these help topic links so that they're subtle and out of the way. And the user would not be prompted or forced to provide feedback. The links are only there to help. A simple example of this can be found in a lot Google's articles. See "Recommended articles" off to the side of this link... google.com/support/webmasters/bin/… –  Steve Wortham Feb 14 '11 at 17:21
    
True, if they are truly unobtrusive and just topics, that may help. It still won't tell Mark where the problems are necessarily. Maybe the app needs some thorough user testing, as opposed to simply seeking feedback. As I indicate below, I think some level of analytics have to be implemented here too. –  jameswanless Feb 14 '11 at 17:25
    
@James - Exactly. Kind of the idea I had is that a pattern of users abandoning a particular page should be a red flag. Also, a user desperately looking for help from a given page should be an indicator that perhaps that page isn't as self-explanatory as it should be. –  Steve Wortham Feb 14 '11 at 17:32
    
Youtube found a similar problem I see: youtube-global.blogspot.com/2009/09/… –  Mark Gibaud Feb 15 '11 at 15:28
add comment

There are many ways you could ask for qualitative and quantitative feedback. As you've employed, ratings systems, open text areas, etc. However, you don't mention how you've implemented metrics and analytics in this process.

You can still use those, but if they are overly skewed positively, then you also need something impartial to evaluate where people are falling down. I'd ask (and measure) a few of the following things:

  • booking completion %'s
  • places where those who don't complete are getting stuck
  • places where the bookings inexplicably drop off

Take a look at the overall completions and then look harder at the % who don't complete and try to determine why. As you say, normal users don't often know what they'd improve, let alone why they are having trouble. If you can start coaxing patterns out of the pages/steps where users are dropping off and bailing, you'll likely start to get some answers.

share|improve this answer
add comment

Good question!

I wouldn't rely to much on the 1-10 scale, as - as you say - it very much depends on the users mind, and without quantificaiton of the scale they'll apply their own measures: What does an 8/10 mean? * "I got lost a few times, but i now I understand, and I like it!" * "Nice, I miss the dancing girls jumping out of pie charts your competition has!"

Rather, offer a few reasonable ratings, plus a free-text field:

Did you like how booking wiht us works?

  • You are the best! Looking forward to my trip!
  • It was easy, but could be improved a little
  • You got me confused
  • It was hard for me
  • God, if your trips are as convoluted a this web site, I'm going to kill myself
  • Can't say.

[ .................... ]
[ .................... ]

Make the free-text field a few lines, and intependent from the choices, so someone feels invited to select "could improve a little" and add a few words what s/he felt was hard.


In addition, run some usability tests. It doesn't have to be anything fancy - the easiest way is to look over someone's shoulder, pick different people - e.g. a fresh intern and even someone who uses the form a dozen times a day. Just watch. Any time you want to grab the mouse, stop yourself and consider how to make the user find what you want him to click.

(I have no problem with the overhead of big testing labs using cameras, trying to reduce the observer effect. However, it's good to get your hands dirty and understand the problems before you ask them).

Another option might be "crowdsourced reviews" such as feedbackarmy offers. I Haven't tried them, but averaging over a bunch of self-declared internet experts might give better feedback than averaging over numbers 1..10.

share|improve this answer
add comment

Ask them what they dont like, you will get specific feedback. Users tend to go into much more detail when asked what they didnt like.

share|improve this answer
add comment

For the information you are trying to find out it sounds like user testing would be a better approach. This doesn't have to be particularly time consuming or expensive. You can pick you users from a wide selection of the public, as many people will have booked holidays/would consider booking holiday online.

Arrange testing with 5 users and sit with them as they perform some scenario you have set. Get them to talk through their thoughts as they go through the process. I think you will get some really interesting feedback that you would struggle to get from a survey.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.