Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

What have you learned from the process of testing prototypes / products with users that you wish you had known before you went into your first workshop? Pitfalls, freak-out moments etc?

share|improve this question
    
if you're looking for a discussion about pitfalls and freak-out moments from workshops, head to the chat! We'd love to have you. UX.SE is a Q&A site based around solving problems. Your question doesn't really match the format. See the FAQ. –  Rahul Jul 26 '11 at 14:51
    
Great question, I wish I could answer it. :( I clicked reopen. –  Glen Lipka Jul 27 '11 at 19:56
    
@Rahul I see where you're coming from, but after running it through the good subjective, bad subjective test I think this question works. –  Patrick McElhaney Jul 27 '11 at 23:51
    
I liked that article. I wish it was applied more. –  Glen Lipka Jul 28 '11 at 15:06
    
Glad to see this back open! –  Matt Rockwell Jul 28 '11 at 16:20
show 2 more comments

5 Answers 5

up vote 7 down vote accepted

Most of my experience with usability tests came from Intuit regarding their public websites. (TurboTax, Quicken, QuickBooks, Intuit, Payroll .com etc) They had people come in and we watched them from behind the mirrored glass. Lots of video cameras, professional usability researchers, eye scanning...the works. Lessons learned:

  1. People are terrible judges of their own behavior, desires, past or future performance. Predictably irrational by Dan Ariely talks about the science behind it. It matches my experience. I literally asked someone how long something took. They replied "It was simple, just a few minutes." 35 minutes is not a few minutes. I showed them the timing and they said it was a lie. The video was a lie. No, people are just bad regarding questions about their behavior.
  2. Watching people is pretty valuable. Actions speak louder than words After a while, I stopped trying to talk to the person or have them think out loud. I just watched what they did. I watched their faces, eyes, hands, feet, mouth. I learned what I screwed up in design by watching people be themselves. Whenever they talked, I ignored it. But whenever they acted, I took note.
  3. It can be done cheaper. I trained all of the beta customers of the first version of Marketo. I did it remotely. I fired up Gotomeeting and made them be the presenter and then I taught them the software and made them do the mouse work. I saw their mouse move and realized it was tracking their eyes. Where is the button? The mouse would wander over the screen. In doing this process, I learned a ton about how our users thought, what they got, what they didn't get. I didn't need fancy equipment and video. I just needed someone who wanted to use the service and gotomeeting. Side benefit: They thought the hand-holding was great and they developed a strong bond with the company.
  4. There is no substitute for creativity. Usability is for exploration, not validation. Talking to 10 people doesn't prove a thing. Executives will use the "findings" to justify their decisions and ignore anything that doesnt support the narrative. Use these kinds of efforts to explore ideas, not to pick the winner. You don't have statistical confidence. Explore areas to improve, explore areas of interest, but don't try to validate something IS or ISNT true. Creativity is the job of the designer and you cant usability or focus group your way there. Use the tool properly and it will give you valuable insight.

I hope this was helpful for you.

share|improve this answer
    
In response to point #1. - Although it took them a long time, it seems as though it was a pleasant experience and they perceived it as being much quicker which is a very good thing as far as user experience goes. –  Matt Rockwell Jul 28 '11 at 15:37
    
+1 Great points here. –  Matt Rockwell Jul 28 '11 at 15:39
    
@Matt: True, but if a competitor came along and could do it much quicker, it would be a significant disadvantage. Additionally, in this case, slowness actually yielded (in larger sample sizes) significant drop off of task completion (and revenue). –  Glen Lipka Jul 28 '11 at 15:49
    
Don't get me wrong, I agree completely, just saying that at least they had a positive experience with it, which can't always be said. –  Matt Rockwell Jul 28 '11 at 15:54
add comment

One: You cannot anticipate every way in which users interact with your prototypes.

Two: You will be disheartened at least once during a testing session.

Three: Check your ego at the door.

Four: It takes longer to analyze the feedback than it takes to gather it.

share|improve this answer
1  
Yes - further to point four, having an idea what you are planning to do with the data you collect, and how, is a good thing to know beforehand. Sounds obvious... –  Roger Attrill Jul 25 '11 at 19:21
2  
Do with the data... you mean, there's a purpose to testing beyond giving the directors a pretty report? Well, crikey. –  gef05 Jul 25 '11 at 19:35
1  
But.. but but.. Directors love spark lines! –  Nic Jul 25 '11 at 20:52
add comment

Long ago, in my first few tests as a moderator, I did a really bad job. I didn't massage data out of users, I just kind of gave them a task and let them run with it.

When I went to a conference with other UI professionals and actually saw how to probe information without leading, I got much better feedback from users. I think it's important to know how to moderate a test - formal, informal, or random - correctly. I wish I had known when I first started.

share|improve this answer
    
Yep. Observing is a skill. +1 –  gef05 Jul 25 '11 at 18:28
3  
Is there any more information you could provide about how to actually go about getting information without leading, or perhaps a link to an article further embellishing this point. I would be interested to read more about what exactly you mean. –  Matt Lavoie Jul 26 '11 at 13:18
add comment

You can't do everything right the first time, so accept that you'll make mistakes and try to learn from them.

Breaks between sessions are not a luxury, you'll need them. When working with a participant you really need to pay attention and concentrate (especially if you don't have a lot of experience yet).

If you can work with a partner and conduct the tests together, it is nice to have one facilitator and one observer (or more).

Don't take critical remarks about the product personally, and don't feel offended. It's natural that - when you've worked on the product or design - you hope that the participants will like it, but in user testing it's a good thing to find out what doesn't work.

share|improve this answer
add comment

That the users don't really want to be there. (Generally speaking, esp without an incentive or reward of some sort).

We held a training seminar on a very complicated product a few months back. This seminar was to teach them how to be somewhat of a master at the product (think like a weekend crash course in something like Photoshop, an easy to use interface, but with exhaustive features.) It also allowed us to observe them using the software and see where they stumbled. Our users are limited, as in they work for the company, needed specific knowledge of a complicated physical product in order to use our software to achieve customized results from the physical product. We started off the sessions with a "dog and pony show" to demonstrate the possibilities of what this software can make the product do. Everyone was impressed and wanted to know how make reality of what they saw.

When it got to learning the software, we thought they would be excited for all of the new features, capabilities, etc. We were wrong. People weren't paying attention and were getting hung up on the most trivial concepts and actions. Even though they were wined and dined all weekend, they still couldn't muster up enough enthusiasm for the product, and most were very disenchanted about the software. We held their hand, step by step, and encouraged them to learn it, but only about 2 people really "got it". Guess which people were the most interested? The two people that "got it".

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.