What have you learned from the process of testing prototypes / products with users that you wish you had known before you went into your first workshop? Pitfalls, freak-out moments etc?

  • if you're looking for a discussion about pitfalls and freak-out moments from workshops, head to the chat! We'd love to have you. UX.SE is a Q&A site based around solving problems. Your question doesn't really match the format. See the FAQ.
    – Rahul
    Commented Jul 26, 2011 at 14:51
  • Great question, I wish I could answer it. :( I clicked reopen.
    – Glen Lipka
    Commented Jul 27, 2011 at 19:56
  • @Rahul I see where you're coming from, but after running it through the good subjective, bad subjective test I think this question works. Commented Jul 27, 2011 at 23:51
  • I liked that article. I wish it was applied more.
    – Glen Lipka
    Commented Jul 28, 2011 at 15:06
  • Glad to see this back open! Commented Jul 28, 2011 at 16:20

5 Answers 5


Most of my experience with usability tests came from Intuit regarding their public websites. (TurboTax, Quicken, QuickBooks, Intuit, Payroll .com etc) They had people come in and we watched them from behind the mirrored glass. Lots of video cameras, professional usability researchers, eye scanning...the works. Lessons learned:

  1. People are terrible judges of their own behavior, desires, past or future performance. Predictably irrational by Dan Ariely talks about the science behind it. It matches my experience. I literally asked someone how long something took. They replied "It was simple, just a few minutes." 35 minutes is not a few minutes. I showed them the timing and they said it was a lie. The video was a lie. No, people are just bad regarding questions about their behavior.
  2. Watching people is pretty valuable. Actions speak louder than words After a while, I stopped trying to talk to the person or have them think out loud. I just watched what they did. I watched their faces, eyes, hands, feet, mouth. I learned what I screwed up in design by watching people be themselves. Whenever they talked, I ignored it. But whenever they acted, I took note.
  3. It can be done cheaper. I trained all of the beta customers of the first version of Marketo. I did it remotely. I fired up Gotomeeting and made them be the presenter and then I taught them the software and made them do the mouse work. I saw their mouse move and realized it was tracking their eyes. Where is the button? The mouse would wander over the screen. In doing this process, I learned a ton about how our users thought, what they got, what they didn't get. I didn't need fancy equipment and video. I just needed someone who wanted to use the service and gotomeeting. Side benefit: They thought the hand-holding was great and they developed a strong bond with the company.
  4. There is no substitute for creativity. Usability is for exploration, not validation. Talking to 10 people doesn't prove a thing. Executives will use the "findings" to justify their decisions and ignore anything that doesnt support the narrative. Use these kinds of efforts to explore ideas, not to pick the winner. You don't have statistical confidence. Explore areas to improve, explore areas of interest, but don't try to validate something IS or ISNT true. Creativity is the job of the designer and you cant usability or focus group your way there. Use the tool properly and it will give you valuable insight.

I hope this was helpful for you.

  • In response to point #1. - Although it took them a long time, it seems as though it was a pleasant experience and they perceived it as being much quicker which is a very good thing as far as user experience goes. Commented Jul 28, 2011 at 15:37
  • @Matt: True, but if a competitor came along and could do it much quicker, it would be a significant disadvantage. Additionally, in this case, slowness actually yielded (in larger sample sizes) significant drop off of task completion (and revenue).
    – Glen Lipka
    Commented Jul 28, 2011 at 15:49
  • Don't get me wrong, I agree completely, just saying that at least they had a positive experience with it, which can't always be said. Commented Jul 28, 2011 at 15:54

One: You cannot anticipate every way in which users interact with your prototypes.

Two: You will be disheartened at least once during a testing session.

Three: Check your ego at the door.

Four: It takes longer to analyze the feedback than it takes to gather it.

  • 1
    Yes - further to point four, having an idea what you are planning to do with the data you collect, and how, is a good thing to know beforehand. Sounds obvious... Commented Jul 25, 2011 at 19:21
  • 2
    Do with the data... you mean, there's a purpose to testing beyond giving the directors a pretty report? Well, crikey.
    – gef05
    Commented Jul 25, 2011 at 19:35
  • 1
    But.. but but.. Directors love spark lines!
    – Nic
    Commented Jul 25, 2011 at 20:52

Long ago, in my first few tests as a moderator, I did a really bad job. I didn't massage data out of users, I just kind of gave them a task and let them run with it.

When I went to a conference with other UI professionals and actually saw how to probe information without leading, I got much better feedback from users. I think it's important to know how to moderate a test - formal, informal, or random - correctly. I wish I had known when I first started.

  • Yep. Observing is a skill. +1
    – gef05
    Commented Jul 25, 2011 at 18:28
  • 3
    Is there any more information you could provide about how to actually go about getting information without leading, or perhaps a link to an article further embellishing this point. I would be interested to read more about what exactly you mean. Commented Jul 26, 2011 at 13:18

You can't do everything right the first time, so accept that you'll make mistakes and try to learn from them.

Breaks between sessions are not a luxury, you'll need them. When working with a participant you really need to pay attention and concentrate (especially if you don't have a lot of experience yet).

If you can work with a partner and conduct the tests together, it is nice to have one facilitator and one observer (or more).

Don't take critical remarks about the product personally, and don't feel offended. It's natural that - when you've worked on the product or design - you hope that the participants will like it, but in user testing it's a good thing to find out what doesn't work.


That the users don't really want to be there. (Generally speaking, esp without an incentive or reward of some sort).

We held a training seminar on a very complicated product a few months back. This seminar was to teach them how to be somewhat of a master at the product (think like a weekend crash course in something like Photoshop, an easy to use interface, but with exhaustive features.) It also allowed us to observe them using the software and see where they stumbled. Our users are limited, as in they work for the company, needed specific knowledge of a complicated physical product in order to use our software to achieve customized results from the physical product. We started off the sessions with a "dog and pony show" to demonstrate the possibilities of what this software can make the product do. Everyone was impressed and wanted to know how make reality of what they saw.

When it got to learning the software, we thought they would be excited for all of the new features, capabilities, etc. We were wrong. People weren't paying attention and were getting hung up on the most trivial concepts and actions. Even though they were wined and dined all weekend, they still couldn't muster up enough enthusiasm for the product, and most were very disenchanted about the software. We held their hand, step by step, and encouraged them to learn it, but only about 2 people really "got it". Guess which people were the most interested? The two people that "got it".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.