4

I'm acutely sensitive to bias in usability testing. For example, I'm aware that task-selection bias is an issue, so I try to keep the tests more open/loose and let participants explore.

I know that there are several other types of bias, as outlined here: http://www.measuringu.com/blog/ut-bias.php

What practical steps have you taken to reduce bias in usability testing?

5
  • Why do you believe bias is higher with remote panels?
    – tohster
    Feb 26, 2015 at 5:08
  • Just to clarify, I don't believe it is higher. It's just a hunch, based on initial experiences. My hunch could be entirely wrong. But I'd still like to mitigate against bias, as much as possible. Feb 26, 2015 at 9:00
  • Ah. in that case unfortunately i don't know how to answer other than to say that if you have a directional hunch then you can a/b test that hunch to see if it's real, or ask whoever is assembling the panel. i haven't had to do anything unusual with remote testing vs on prem.
    – tohster
    Feb 26, 2015 at 10:22
  • You're right, I think I was confusing the issue by introducing my own hunch. I've clarified the question to make it more general, and removed references to remote testing. Thanks for making me re-think this :) Feb 26, 2015 at 11:08
  • 1
    That's a good list on the measuringu website link.
    – PhillipW
    Mar 2, 2015 at 9:53

4 Answers 4

3

Aside from what is on that list, when I conduct usability tests I:

  • Try to ensure that I give identical instructions to each participant.

When people are doing tests remotely, it's easier to do this as you can provide written instructions. In person, however, I try extra hard to avoid going "off-script" and potentially leading anyone towards answers I might want.

  • I also avoid indicating myself or my department in the product, to avoid making the interviewee feel like they would be offending me by giving honest feedback.

Otherwise, I also try to conduct tests in a way that avoid the issues on the list: I avoid taking notes (record everything), I avoid asking specific tasks, or if I do ask for a specific task, I give a range of them.

3
  • Interesting point about avoiding taking notes. I do this too, for practical reasons, but can you explain a bit more about why you feel this also helps reduce bias? Feb 27, 2015 at 10:16
  • I only take notes of stuff I want to pick up in the moment with the user. This means I can concentrate more on my interaction with the user. Actually recording what is going on I leave to the technology.
    – PhillipW
    Mar 2, 2015 at 9:52
  • I agree. The measuringu article makes another good point, of course, that if note-taking is too overt, it may cause the participant to change their behaviour -- I remember taking my driving test and thinking "he's writing something down, what did I do wrong?!" Anyway, I'm picking this answer over other good answers because it had a couple of clear, practical tips. Mar 2, 2015 at 11:13
2

“All the world's a stage, and all the men and women merely players”

I have conducted usability testing in a number of environments both controlled and uncontrolled and I genuinely believe that there is no silver bullet to deal with bias, but you do have to take precautions and consider impact on test results.

Erving Goffman who was a very influential sociologist wrote a book in the 60s entitled The Presentation of Self in Everyday Life in which he describes the theatrical performances that take place in face-to-face interactions. Sorry for the long quote but it is so relevant:

When an individual comes in contact with another person, he attempts to control or guide the impression that the other person will form of him, by altering his own setting, appearance and manner. At the same time, the person that the individual is interacting with attempts to form an impression of, and obtain information about, the individual … participants in social interactions engage in certain practices to avoid embarrassing themselves or others. Society is not homogeneous; we must act differently in different settings. This recognition led Goffman to his dramaturgical analysis. He saw a connection between the kinds of "acts" that people put on in their daily lives and theatrical performances. In a social interaction, as in a theatrical performance, there is an onstage area where actors (individuals) appear before the audience; this is where positive self-concepts and desired impressions are offered. But there is, as well, a backstage – a hidden, private area where individuals can be themselves and drop their societal roles and identities.

Taking precautions to minimize onstage performance:

Every testing situation is unique and depends on who you are testing with but as a rule you need to account for situations that are more likely to have an impact on "onstage performance" for example by:

Preparing test participants and reassuring them: This allows you to inform and reassure testing participants that the usability testing session is not meant to test their abilities but rather the product performance (should be part of a formal testing protocol that also has a confidentiality agreement).

Configuring usability testing environment: This allows you to assess the testing environment and decide on optimal layout for testing to take place. you should be able to clearly observe what is happening while gradually easing yourself out so users don't focus to much on the fact that they are being observed.

Analysis and focusing on the "backstage":

As results are bound to have some onstage issues, you need to factor those in for both yourself and test participants. The way I go about this is by constantly questioning user actions at each action point. This translates to two list, first: is "why the user is doing this" and second: "because" then i list all possible triggers along with explanations till I have exhausted all possibilities, which should in principle allow me to spot any onstage performance while zooming in on the Backstage stuff:)

Remote usability testing bias

This one is a bit tricky, I think you might be right about suspecting a bias of some form but you could mitigate this by checking and reviwing the recruitment process,demographics,type of reward,if users have participated in many remote testing sessions before or not! my last point, they are becoming professionals and thus not right for the job :)

1
  • Great point about testers becoming professionals. I think this is maybe what I am sensing in the well-known remote usability testing services that I am using. The participants' "performance" seems too polished. In my gut, I don't trust it. Mar 2, 2015 at 11:02
1

I came across an issue with meta programming in one early test I did.

I was asking the following question:

A form on a web page needs to have a way of clearing or submitting the data – traditionally this is handled with two buttons. One says ‘Cancel’ and the other says ‘Submit’. Of the two buttons, One is on the left and the other on the right. Which one is the ‘Cancel’ button?

I had tried to select candidates who were inexperienced with computers or smart phones in order to get a platform-independent view. The results seem to favour putting the cancel button on the left.

Afterwards, I rechecked the results and suddenly noticed the way the question was worded. The word “one” appears three times and is used twice in connection with the ‘Cancel’ button, the concept of cancelling is referenced four times but the concept of submitting is only referenced twice and the order that the two process are mentioned is constant: “clearing”/”submitting” and “cancel”/”submit”.

A few days later I went out testing again but this time with the following question:

A form on a web page needs to have a way of clearing the data – traditionally this is handled with two buttons. Which is the ‘Cancel’ button?

The result was much more closely balanced. - The first question had led the candidates but the second put the choice more in their hands.

0

One issue which comes up when doing testing is how not to give away the answer while asking a question:

For example - There's a button which users are supposed to press which carries out the next part of the process. For some reason they don't get this and are happily going off and clicking on other buttons on the page. So as a tester you need to find out why they don't click it. If you say 'why don't you click on the "the button" you've told them that that's what they are supposed to do (and they most times will just say they don't know)

... So really what you need to do is think of a way of giving them a hint that they're not doing it right, but still leave them actually solving the problem so that you can understand from the Thinking Aloud how they actually solve it.

And you've got to do this live as you never know when this kind of thing is going to come up.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.