How do you control for a user's state of mind when asking for quantitative feedback?

Question

So we have a site that processes the booking of a travel tour over 4 or 5 pages in a wizard format.

A while ago I put a feedback section on the final page of that process. This consisted of a rating of your "booking experience" on a scale of 1-10 as well as a free-text comments section.

Now our site has pretty much been UX-designed by developers (it has been graphic-designed by a designer so looks pretty) which means it's not as usable as it could be, and we get a good few of the basics wrong. I expected average to above-average scores from the 1-10 rating.

Over 20 thousand ratings later and a huge proportion of them are 9 and 10 out of 10. A lot of people seem to think it's flawless!

Now I'm aware that the average user wouldn't know what improvements to make even if they could say that a particular page/section was not very easy-to-use, so maybe they have that thought and when they can't think of anything that is obviously improvable they decide to vote a 9 or 10.

But I'm also aware that people tend to blame themselves when they can't get the product to do what they want. Consequently, if they are at all frustrated during the booking process and then do finally make it to a page that has a nice big "Booking Confirmed" on it, they are instantly overloaded with utter euphoria and the endorphin-rush is ruining the objectivity I need for accurate feedback data.

How do I control for the users' positive state of mind ("Yay I've just made a booking!") when asking for a feedback rating?

Answer 1

Maybe the feedback rating is not what you need in this case.

It sounds like you're interested in discovering potential problems with the process. But those who actually complete the wizard aren't having problems.

So instead, what if you create links to appropriate help topics off to the side of every page? Those who are having problems will tend to click through the help. And then through whatever analytics system you're using, you can then review trends where people are having problems and which specific help topics they're looking at. As time goes on, you can use these insights to improve both the help and the wizard itself.

Answer 2

There are many ways you could ask for qualitative and quantitative feedback. As you've employed, ratings systems, open text areas, etc. However, you don't mention how you've implemented metrics and analytics in this process.

You can still use those, but if they are overly skewed positively, then you also need something impartial to evaluate where people are falling down. I'd ask (and measure) a few of the following things:

• booking completion %'s
• places where those who don't complete are getting stuck
• places where the bookings inexplicably drop off

Take a look at the overall completions and then look harder at the % who don't complete and try to determine why. As you say, normal users don't often know what they'd improve, let alone why they are having trouble. If you can start coaxing patterns out of the pages/steps where users are dropping off and bailing, you'll likely start to get some answers.

Answer 3

Good question!

I wouldn't rely to much on the 1-10 scale, as - as you say - it very much depends on the users mind, and without quantificaiton of the scale they'll apply their own measures: What does an 8/10 mean? * "I got lost a few times, but i now I understand, and I like it!" * "Nice, I miss the dancing girls jumping out of pie charts your competition has!"

Rather, offer a few reasonable ratings, plus a free-text field:

Did you like how booking wiht us works?

• You are the best! Looking forward to my trip!
• It was easy, but could be improved a little
• You got me confused
• It was hard for me
• God, if your trips are as convoluted a this web site, I'm going to kill myself
• Can't say.

[ .................... ]
[ .................... ]

Make the free-text field a few lines, and intependent from the choices, so someone feels invited to select "could improve a little" and add a few words what s/he felt was hard.

---

In addition, run some usability tests. It doesn't have to be anything fancy - the easiest way is to look over someone's shoulder, pick different people - e.g. a fresh intern and even someone who uses the form a dozen times a day. Just watch. Any time you want to grab the mouse, stop yourself and consider how to make the user find what you want him to click.

(I have no problem with the overhead of big testing labs using cameras, trying to reduce the observer effect. However, it's good to get your hands dirty and understand the problems before you ask them).

Another option might be "crowdsourced reviews" such as feedbackarmy offers. I Haven't tried them, but averaging over a bunch of self-declared internet experts might give better feedback than averaging over numbers 1..10.

Answer 4

Ask them what they dont like, you will get specific feedback. Users tend to go into much more detail when asked what they didnt like.

Answer 5

For the information you are trying to find out it sounds like user testing would be a better approach. This doesn't have to be particularly time consuming or expensive. You can pick you users from a wide selection of the public, as many people will have booked holidays/would consider booking holiday online.

Arrange testing with 5 users and sit with them as they perform some scenario you have set. Get them to talk through their thoughts as they go through the process. I think you will get some really interesting feedback that you would struggle to get from a survey.