How to design user evaluation tasks without leading people to 'the answer'?

Question

By way of example: on the site I am testing I have a page that shows a piece of information; and a side panel that shows related / similar information from other areas of the site. How do I test whether the users in my experiment notice the sidebar without telling them it's there?

My best effort thus far is just to use different words from the header on the sidebar, e.g. "How would you find other information RELATED to this?", when the sidebar header says, "See more...".

Any other tips? I'll give an imaginary second upvote to anyone who can give me generalisable advice, because this isn't the only situation.

Answer 1

You could use a specific use case instead of a general one and observe how your test subjects uses the sidebar as a tool to complete that task.

For example your main article might be 'A guide to good fish restaurants in Manhattan'. Your test case might be 'Where would you go to find articles about good steak restaurants in Manhattan?'

Answer 2

Your own strategy sounds pretty good. If this is a continual issue for you though, you may want to look into a low-cost eye tracking solution. Eyetracking is much less invasive than it used to be, and much cheaper as well. I wouldn't be surprised if there's some open source eyetracking software out there that will run on a cheap webcam. A cheap solution won't have great fidelity, but you don't really need it when you're only trying to discriminate between large regions on a page.

Answer 3

There are a couple of general tactics I use to design non-leading research.

1. Feedback feedback feedback! This is the single most important thing, because it's really easy to forget how much more you know about the problem you're trying to solve than the user does. I encourage my team members to call me out when I'm being leading and give them positive reinforcement when they do. When I first started out, I also recorded all of my research sessions and would watch them later to try to catch myself.
2. Script critical questions / instructions and have an outline for the session. When you are struggling to construct an instruction on the spot you might be distracted and accidentally give more cues than you intend.
3. Non-leading, open-ended questions. Questions should never be yes/no or imply a dichotomy. The most leading question you could ask is "There was a sidebar on the page you just saw. Did you notice it?" Less leading would be "What do you recall about the page you just saw?" Even less leading would be "What did you do to try to accomplish [task]?"
4. Silence is your best friend. Every time you open your mouth, you give more cues to the user. When you get in the habit of asking open-ended, non-leading questions, you'll find that you need to talk very little, except to move on to the next topic or prompt somebody to dive a little deeper with a question like "what's good about that?" or the ever-useful "why?"
5. Carefully consider the sequence of revealed information. Often you'll need to reveal some of your objectives to dive a little deeper, and you need to make sure you do it in the right order.

I recently designed a telephone interview to see if customers who had signed up for a specific offer were influenced by a particular component of that offer. The team had the question - did customers notice that component? Their thinking was that if customers hadn't noticed it, the solution would be to make it more prominent. Experienced researchers should be hearing alarm bells at this point.

The interview I designed started out very general - understanding their motivations for looking for the product, how they used it, what they liked about it. I gave the interviewer strict instructions not to mention the specific component until the very end of the interview - if the customer noticed that component and it influenced their decision/use, then it would have come up organically in the open-ended discussion of their motivations and use. At the very end, if it never came up, we'd ask if they knew that specific component was part of the offer.

As it happened, it almost never came up organically in the discussions, so we almost always had to ask directly. We found that some were and weren't aware, but it was far more powerful for the team to have started with that open-ended context to understand why customers never noticed the offer component - it clearly wasn't important to their decision to buy! If we'd just asked if they knew about the offering component and if it had an influence on their decision, that would have raised all sorts of churn in the team about how to make the offer more noticeable.

Answer 4

Give users tasks with simple goals (e.g. "find all the information related to X") before they start. If they don't use that part of the interface, then they didn't find it! Any direction is skewing your data. Yes, you can make it more challenging by not letting them visually scan the page for words in the task--but sometimes users will miss it even if you do!

Once they have tried and failed to find your feature a few times, you'll know it is insufficiently discoverable and work to fix that problem. During that work, you can of course direct people to it if you want to work on the design of the feature in addition to its discoverability.

But it's important to sit there and let the user painfully pass over the feature, all while the developers and designers are squirming in discomfort, just to make sure everyone really understands how hard it is to use a new interface and how hard it is to make good, discoverable controls.

Answer 5

Having written test plans in the past to avoid the problem of prompting I've aimed to design questions in the form of stories that cover the reason the site exists in the first place. This maybe the business goals or use cases or a mix of both. The plan being that the participant has a short story to replay whilst using the site/prototype/wireframe this has proven successful in avoiding the need to prompt.

Where participants start to ver off in the wrong direction I let them go as this is a real scenario and we can observe how they go about rectifying their journey so they are back on track.

Often I've found that if a navigation feature has not been seen it is a regular failing that needs addressing and once the scenario has played out I'd go back and ask questions about the problematic areas. Did you see this…, where you aware of that link…, are these links relevant for continuing your journey…. On many occasions the links have been missed or more regularly they have purposely been ignored as they were felt to be insignificant through poor labelling or bad positioning.

Answer 6

Agree that eye tracking could be the solution, but generally isn't cheap. I know there are various tools out there that use algorithms to produce eye tracking/heat maps, but I have no idea how reliable they are. Also eye tracking only shows that the user looked at that area of the screen, not what information they got from it or what actions it drove.

I think your approach sounds ok. Giving the user basic scenarios where they have the potential to use the side bar and recording whether they do or not. Maybe at the end of the session you could ask the user if the side bar was of any use, but note the feedback you get from this will be limited to the user being consciously aware of why they did what they did.