What do you wish you had known before your first ever user evaluation workshop?

Question

What have you learned from the process of testing prototypes / products with users that you wish you had known before you went into your first workshop? Pitfalls, freak-out moments etc?

Answer 1

Most of my experience with usability tests came from Intuit regarding their public websites. (TurboTax, Quicken, QuickBooks, Intuit, Payroll .com etc) They had people come in and we watched them from behind the mirrored glass. Lots of video cameras, professional usability researchers, eye scanning...the works. Lessons learned:

1. People are terrible judges of their own behavior, desires, past or future performance. Predictably irrational by Dan Ariely talks about the science behind it. It matches my experience. I literally asked someone how long something took. They replied "It was simple, just a few minutes." 35 minutes is not a few minutes. I showed them the timing and they said it was a lie. The video was a lie. No, people are just bad regarding questions about their behavior.
2. Watching people is pretty valuable. Actions speak louder than words After a while, I stopped trying to talk to the person or have them think out loud. I just watched what they did. I watched their faces, eyes, hands, feet, mouth. I learned what I screwed up in design by watching people be themselves. Whenever they talked, I ignored it. But whenever they acted, I took note.
3. It can be done cheaper. I trained all of the beta customers of the first version of Marketo. I did it remotely. I fired up Gotomeeting and made them be the presenter and then I taught them the software and made them do the mouse work. I saw their mouse move and realized it was tracking their eyes. Where is the button? The mouse would wander over the screen. In doing this process, I learned a ton about how our users thought, what they got, what they didn't get. I didn't need fancy equipment and video. I just needed someone who wanted to use the service and gotomeeting. Side benefit: They thought the hand-holding was great and they developed a strong bond with the company.
4. There is no substitute for creativity. Usability is for exploration, not validation. Talking to 10 people doesn't prove a thing. Executives will use the "findings" to justify their decisions and ignore anything that doesnt support the narrative. Use these kinds of efforts to explore ideas, not to pick the winner. You don't have statistical confidence. Explore areas to improve, explore areas of interest, but don't try to validate something IS or ISNT true. Creativity is the job of the designer and you cant usability or focus group your way there. Use the tool properly and it will give you valuable insight.

I hope this was helpful for you.

Answer 2

One: You cannot anticipate every way in which users interact with your prototypes.

Two: You will be disheartened at least once during a testing session.

Three: Check your ego at the door.

Four: It takes longer to analyze the feedback than it takes to gather it.

Answer 3

Long ago, in my first few tests as a moderator, I did a really bad job. I didn't massage data out of users, I just kind of gave them a task and let them run with it.

When I went to a conference with other UI professionals and actually saw how to probe information without leading, I got much better feedback from users. I think it's important to know how to moderate a test - formal, informal, or random - correctly. I wish I had known when I first started.

Answer 4

You can't do everything right the first time, so accept that you'll make mistakes and try to learn from them.

Breaks between sessions are not a luxury, you'll need them. When working with a participant you really need to pay attention and concentrate (especially if you don't have a lot of experience yet).

If you can work with a partner and conduct the tests together, it is nice to have one facilitator and one observer (or more).

Don't take critical remarks about the product personally, and don't feel offended. It's natural that - when you've worked on the product or design - you hope that the participants will like it, but in user testing it's a good thing to find out what doesn't work.

Answer 5

That the users don't really want to be there. (Generally speaking, esp without an incentive or reward of some sort).

We held a training seminar on a very complicated product a few months back. This seminar was to teach them how to be somewhat of a master at the product (think like a weekend crash course in something like Photoshop, an easy to use interface, but with exhaustive features.) It also allowed us to observe them using the software and see where they stumbled. Our users are limited, as in they work for the company, needed specific knowledge of a complicated physical product in order to use our software to achieve customized results from the physical product. We started off the sessions with a "dog and pony show" to demonstrate the possibilities of what this software can make the product do. Everyone was impressed and wanted to know how make reality of what they saw.

When it got to learning the software, we thought they would be excited for all of the new features, capabilities, etc. We were wrong. People weren't paying attention and were getting hung up on the most trivial concepts and actions. Even though they were wined and dined all weekend, they still couldn't muster up enough enthusiasm for the product, and most were very disenchanted about the software. We held their hand, step by step, and encouraged them to learn it, but only about 2 people really "got it". Guess which people were the most interested? The two people that "got it".