Getting honest feedback on usability tests

Question

I work in a small company where I'm the only UX person on the team. We're in the process of conducting usability testing on an important section of the system that involves transforming of existing client data into a new format. There is no way to undo this change, so we want to ensure we iron out as much of the kinks as we can through testing.

One of the dangers of the designer doing the testing is the possibility that the user telling what they expect I want to hear instead of what's actually on their mind.

In the past, I phrased open ended questions to avoid leading the user and attempt to provide multiple options so there is no one "right" answer. We're now at the final stages where we're down to a single solution. So I can't use that approach any more.

I've read in some books you can lie and say you're just running the test and are not involved in project and so the testers can be blunt about their criticisms. This is not an option in my case because the users know I'm the only designer.

What are some other techniques that will help in getting honest feedback from testers?

Answer 1

Combine two research methods

Start with usability testing, for the reasons that @Kit-Grose gave. Have a quick glance at the illustration in this NN/g post about UX-research methods to understand which methods can give you insights into user performance, and which methods can survey their opinions and ratings.

Then follow that research with a method that gets people to talk freely, after the usability-research session. I've seen Microsoft apply this method at the end of U-test sessions:

1. Show the participant a pageful of line drawings drawings of random objects and symbols, arranged like tiles. (See the tip, below, to gather suitable images.)

2. Ask the participant to "Identify three pictures that best relate to the activities you've done." After they choose three images, ask them to "Please explain why those drawings are best."

It works because…. The function of the line drawings is to distract the participant from the fact that they're criticizing you. Instead, they're justifying a choice of theirs, which is a very different activity, and emotionally much easier to do. When participants tell you why a given image applies, they'll be telling you what they feel and think of the experience. As they explain why they chose their three images, you'll be surprised at what the participants say and imply about your product or the experience of trying it.

Of course, the actual choice of images is irrelevant to you. What you want to note is what participants say about them.

This method takes 2 or 3 minutes of participant time, near the end of the research session.

Tips for selecting suitable images

• You can make your own page of small images by selecting a varied sample from an image search such as this for small line drawings or such as this for 100 line drawings.
• Choose simple objects, not too much detail. Select images such as a clock, an ambulance, a log, a squirrel, a hand, a book, a calculator, a smart phone, an apple, a door, a dog, a ruler, an open box, a car, and so on.
• Stay away from images that are obviously fishing for compliments, such as a heart.
• Ensure the images have the same value (lightness or darkness). Here's how: as you squint at the page so the images become blurred, no image should stand out because of its relative lightness or darkness in comparison to the other images on the page. Avoid high contrast:

Answer 2

You shouldn't really expect to get explicit feedback from a usability test (honest or otherwise); that's what focus groups and other such things are for.

Usability testing is a way to test outcomes, not opinions. It's a great way to see if the changes have a noticeable improvement (or at least equivalence) when it comes to users performing a given set of tasks. It's not really a good idea to solicit feedback from participants in a usability test—you want their feedback to come through organically while executing the task at hand. You aren't asking for the participants' expertise as designers, rather their domain knowledge at the task they're trying to perform.