How to allow for 'exploration' in your user testing?

Question

In a lot of user testing sessions I've been in a common task is to test the effectiveness of a particular icon or navigation control.

The typical scenario:

tester: How would you accomplish task x?

testee: They stare at the screen for a while because they are in 'test mode' and feel like they need to narrow it down to one and only one option before answering.

The failing here, in my opinion, is that this excludes the natural discovery process that many people use when looking at a new app--especially on touch devices.

For example, many apps are heavily icon-driven (instagram, Facebook, Flickr, etc) and even for a visual designer such as myself, I often don't sit there and attempt to decipher each icon before tapping it. I just start tapping and playing with the app.

My questions:

Is 'playing/discovering' a common technique for users to figure out how to navigate an app?

If so, have you figured out ways to accommodate this in your user testing scenarios?

The situation I'm trying to avoid is one where we do some user testing, and we find out that a particular icon wasn't immediately identifiable by the user when asked. We then consider that a problem, even though I hypothesize that outside of the user test, people would just tap on it then immediately know what the icon is. In other words, this is an issue with the test--not the actual icon. Alas, we then focus time and energy on changing an icon instead of fixing more pressing usability issues.

UPDATE: I clarified the scenario to make it more task-focused rather than feature-focused. The feedback about making sure it is task-focused is valid and appreciated. That said, I think the concern is still there in terms of making sure exploration is encouraged.

Answer 1

There are a few ways to allow for exploration in your user testing.

When I'm conducting research with a prototype or application, my first task is almost always something along the lines of, "What do you think about what you see here?" It's an open-ended question before we talk about anything else. It helps set the participant at ease. It gives the participant a chance to share their first impression, and tell me if they see something that they don't understand or that they find intriguing. This works whether you're testing a full application or just showing low-fidelity wireframes/prototypes. You can, if you have time and if the fidelity of what you are testing allows, let the participant explore the experience a bit, so long as you are able to get them back to a standard starting point for the research that you are conducting.

My last question, if time allows, is usually something along the lines of, "Now that you've had a chance to use this a little bit, would you like to check something else out?" It allows the participant to explore further if they're so inclined. I also give them the opportunity to ask me questions, although anything future-looking that we didn't discuss is usually an opportunity to gather more data ("how would having [that feature] impact what you do today?").

I also make sure to phrase my tasks such that they're real-world tasks, not just "go find the feature". Good phrasing of tasks means that you are not leading the user towards a specific path, which gives your user some ability to explore the application as they try to determine how to complete the task. If you are especially interested in having them complete the task in a certain way and they go down a different path, you can simply say, "that is one way to do that, can you find another way?" This gives you much richer data, because you learn how participants naturally try to accomplish the task and whether they can find the intended path.

Answer 2

Yes, definitely, playing and discovery can be an important part of a test session. If it's insight you're after, a self-directed task is great for many reasons.

Helps the participant to feel at ease.

One way I have tried is to have 4 normal tasks, but the very first task is self-directed.

Given you have this app in front of you, what are curious about? This is where I invite you to explore before we begin.

You would think that a person might feel self-conscious about this, and decline the offer to explore, but that has never happened to me. It also serves to take the pressure off the test situation, and because the person gets to satisfy their curiosity right off the bat, they get something right away (other than the snacks) and are more giving during the test session. I highly recommend starting your test with a self-directed task initiated by your user.

The only drawback is that they may come across something you were trying to wait to roll out until later, but in real life this will happen anyway.

Reduces the need to explain what the app is.

You don't need to explain much -- you can start by stating,

Today you'll be helping us evaluate "Digeridon't", a music app for iPhone and Android. Looking at the interface, what do you think this app does?"

By asking a few key questions at the outset, you can also test the way the app is perceived. Can they guess that "Digeridon't" sends special silencing drones after street musicians? If not, the team may need to re-tool the look of the app.

Exercises the app past the boundaries of the test

Users will explore at will. You will find things you wouldn't find in even the best-designed test.

Answer 3

Basic Rules:

• Trying things out is what users do.

• Reading, and following structured instructions on screen is what they don't do !

Answer 4

I usually set a very informal environment before the testing begins. Just spend some time learning about the user on a face to face basis. It helps put them at ease and also once the camaraderie is established, I ask them to play around with the prototypes ( if any ) The word play is quite effective as they see it as a game where they can find the challenges. I can later request them to be really honest with their experience so that it helps me the best. Surprisingly, they open up and really give honest try.

I do not ask them questions until they are done with exploring. If they ask for help on anything, I try not to give them direct answers but guide them so as to see how much they can accomplish with as little help as possible. Many people stumble at this point saying, I feel stupid I don't know how this works and I remind them it's the app that needs to be better, not them. Always be keen on observing the participants when they explore. There are many ' blink-and-you'll-miss' moments that can give you real insights about user research.

Always ask open ended questions about their experience with the app, something along the lines of, what are your thoughts about this? /What do you think this is for?/ How was your experience?/ Would you like to explore a bit further?

Never talk about a particular feature, rather if you do, keep it to their experience with it. Try not to explain what the protoype does at any time. Our aim is to figure out how much of the design is intuitive.

Answer 5

This sounds like a good example of how NOT to do user testing. The aim of user testing is not to ask the user what features they like or want directly. The aim is to see if they can carry out a task and, ideally, that task has been tailored during the warm up conversation. All user testing needs a warm up period where you get to know the user, what they do, why they do it and some general overall questions before launching into testing tasks not features.

If a tester ever asked

How would you find the x feature?

I would have a word with them! Feature is not how a user thinks of the world - they think in terms of finding things out and getting things done.

With visual design the temptation is to ask questions about it directly. A question like 'What do you think of this icon' has not place in user testing. Instead the icon should be part of the user experience (that is the flow of the application) and if it is not working then putting users through a task where identifying that icon as a step will quickly tell you if it's a problem. For example the 'burger' icon used on phone, tablet and sometimes desktop sites is infamous for being a stumbling block to users as it's used in too many different ways for a user to understand what is under it. Putting the world 'menu' under it helps and ties in to the guideline that ever icon needs text to support it.

So in short the tester should be asking them to do things and be told that 'some things we're not sure about and we want to know if we've done them correctly', or something like that. The user then doesn’t mind if they fail at a task as they've caught you out, not the other way around. This is why some user research training is vital for all testers! It's about tasks not features.