16

I realise 'went wrong' is not the correct term here. We're not testing the user, we're testing the site. (Hence the quote marks). Anyway:

Take this hypothetical scenario: You're running a usability test and one of the scenarios is to see if the participant can find a way to get in contact with your company in a way that won't have people in their office listening in on what they're contacting you about. (Basically a scenario geared around finding out how easy it is to find the Live Chat option on the website).

However the participant looks around a few areas of the site and finds an mobile telephone number field in the site footer and states that they would probably just send a text to that number.

Well, that obviously isn't the result you were hoping for, but it's a very valid test anyway as it suggests that you've not presented the Live Chat feature in an intuitive way.

However, because they didn't find the section of the site you were after you may feel like you want to find out why they didn't click on the obvious 'Begin a Live Chat' button that's been there all along.

But should you point that out to them?

  • The benefit of doing so is that you can possibly get some additional feedback as to why it wasn't noticed.

  • The negative of doing so is that you may make the participant feel a bit stupid for not noticing it (which they obviously aren't; it's your site design at fault, not them).

So what to do? Point these things out whenever they get to the end of that particular scenario? Soldier on with the testing and reveal all the bits they missed at the end? Don't mention any of these things to them at all?

Improve this question
4
  • I suggest to change the question title to something like "Should you have participants redo parts of usability tests by walking them through the first few steps if they 'went wrong' there?" The way it is written, I expected a question about whether or not to inform participants about the expected/hoped for solutions for the tasks without any effect on the stored answers, just to satisfy the participants' desire to learn about the "correct" solutions.
    – O. R. Mapper
    Jul 31, 2014 at 6:43
  • @O.R.Mapper: That wouldn't really be 'redoing' the usability tests though, as the test revolve around completing various scenarios in whatever means they wish. However I have seen that participants recognise they have clearly missed something but don't want to say that (perhaps out of embarrassment?) so I'm considering giving them a bit of closure by pointing out the more optimal routes. That wouldn't impact the already provided answers as I would wait until the end of the session to give such feedback.
    – JonW ♦
    Jul 31, 2014 at 7:55
  • "you can possibly get some additional feedback as to why it wasn't noticed" sounds to me like you would be gathering additional answers. Anything that may be considered in your evaluation means "impacting the answers".
    – O. R. Mapper
    Jul 31, 2014 at 7:57
  • @O.R.Mapper: Perhaps it's just a terminology point of view. With these usability tests I will run through some scenarios and record the results (i.e. 'answers') to those scenarios. I would also ask some general feedback questions and gain additional information at that point. However the responses to these additional questions would not change how I recorded the scenario run throughs. Two sets of 'answers', I suppose.
    – JonW ♦
    Jul 31, 2014 at 8:09

4 Answers 4

Reset to default
22

Asking someone why he didn't notice something isn't likely to provide useful information. People will confabulate a response for the sake of their own mental consistency without even realizing they've done so.

We can only focus on a few things at once; if we don't notice something, it's because we were paying attention to something else. He didn't notice the gorilla because he was looking at the basketball players. She didn't notice the Live Chat because she was looking at the telephone number. Seeing what someone did notice implicitly tells you why that person didn't notice something else.

At the end of each task, we ask participants "Was there any other way you could have completed the task?"

This question lets you see whether the user is able to discover the alternate way you had hoped they would find without explicitly pointing it out to them. Even if they don't see any alternate way of completing the task, this still gives you information about what they are seeing so that you can graphically de-emphasize that in addition to emphasizing whatever you do want to be found.

Improve this answer
1
  • 2
    +1 for don't ask why, they'll tell you anything seen several tests where this happened. The tester is not at all qualified to analyze his own actions! That's your job and you are many times more objective than the tester himself. I've seen testers complaining about graphics, when there problem was with sound, or colors, when another shape helped...
    – Falco
    Jul 31, 2014 at 11:50
3

I would not say there is a definitive answer, rather it would depend on how you conducting the test and what type of data you are looking for.

Tasks vs. Scenario

Tasks would be quick and to the point, where participants don't necessarily need at background information. Having a series of very discrete tasks, with little to no relationship to each other, would allow of more immediate querying of where they "went wrong" after a task.

Scenarios provide rationale and context to the user, setting the stage for a story to be told. Asking the user to break out of the story to explain what "went wrong" could affect at least their immediate reactions to the next task in the scenario, as they work to get back into context.

As tasks can more easily be broken with immediate queries, a more scenario based test is what becomes more tricky...

Qualitative vs. Quantitative

What type of data are you looking for?

Success metrics for quantitative test data include the number of successful tasks completed and the number of successful/unsuccessful steps within a task. While to say the reason behind what "went wrong" in quantitative data "doesn't matter", its weight is diminished. If you are looking for strict quantitative metrics, breaking in to ask is less helpful.

Collection of qualitative data through observation and understanding includes multiple insights. Included is what problems where encountered - not knowing where to go, what to do next, finding what they were looking for, and could they recover from errors?

In qualitative data collection, asking where the user "went wrong" is much more important. But when?

Immediate vs. Post-Test

Once the facilitator has allowed the user to flounder for a bit, a period that is unique to each test, its time to get them back on track. When the user is guided back to the correct point the facilitator has the choice to immediately break in, or to let the user continue.

The facilitator can go through several attempts, asking the participant to "think aloud" to get an understanding of why they are missing the button. A good researcher might find more meaning in a participants verbal walkthrough, than if the participant is just asked "why did you not click that?" Participants might not be able to see exactly how they "went wrong" and may not be accurate about it (intentionally or unintentionally).

You've not outright asked the participant why they "went wrong", reducing the potential for "feeling stupid", but you can still pull out the information you were looking for.

Asking in a post-test situation, if you are seeking qualitative data, suffers from a participants potential inability to remember why. If the testing session is too long, and the "error" was early on, the user may not remember what happened. Additionally, their memory will be muddled if they had to be shown the button in order to reach a new task; they will likely remember being shown the button more than why they missed it.

If a post-test questionnaire is provided, it just makes sense to ask for an explanation of what "went wrong". But the answer needs to not be taken purely at face value.

Immediate Discovery AND Post Rationale

In the event the facilitator needs to "rescue" the participant, discovering what went wrong is more useful then asking what went wrong. Having the participant work through, and verbalize, their task flow demonstrates their mental model to the facilitator - who can then understand what "went wrong". Simply asking allows the participant to think about it, and change their answer.

Although subject to memory and ego, ask the participant for rationale after the test (or at logical break points within the test, such as between scenarios) for their take on the situation. Answers can still yield valuable insight into why the "correct" action was missed.

Improve this answer
2
  • There is also an option between immediate and post-test; post scenario. Usually my usability tests, especially if it's just guerrilla testing, is just 3 or 4 short scenarios, so I would look to get a bit of feedback from them immediately following completion of each scenario. It is at that point I would wonder whether or not to ask about the options they may have missed during that particular run.
    – JonW ♦
    Jul 31, 2014 at 8:07
  • Agreed. Still working to 'discover' what went wrong during a scenario and breaking for a questionnaire between scenarios (or other "logic break point within the test", from the last paragraph) is - IMO - preferred. I've done multi-day tests before, in which giving the participants a questionnaire and asking them what they thought about something 2-days ago would have been pretty useless.
    – Nicholas Pappas
    Jul 31, 2014 at 15:26
2

I'd probably point it out to them, but I'd phrase it very carefully, putting the fault on the website design: "Did you notice this button up here? Many people miss it, do you think it just isn't noticeable enough or did you disregard it for a reason?"

Improve this answer
1

But should you point that out to them?

Yes. In a kind way, if its part of the functionality of the site to be tested. This also allows you to probe a bit as to whether they noticed it but didn't use it (and why) or whether they just didn't notice it in the first place

This is also just part of a much bigger problem with testing: you'll have a user for a certain period of time so you are trying to maximise the usage of that time.

Much as you don't want to interfere with what the user is doing there will be times when you watch them wander off in completely the wrong direction on a site and you can see them blowing the rest of the testing period being lost in some obscure corner of a website with you learning nothing and them getting frustrated.

There are therefore always times when the tester has to step in and 'rescue' them so that the testing period can be used working through the other scheduled tasks.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.