18

I'm pushing usability testing within my workplace by performing some Steve Krug style usability tests, which have gone down so well that the product owner now wants me to add some marketing questions.

It's her budget, and the questions are perfectly sensible, but I need to explain what usability tests are good at and not so good at. It seems to me that by concentrating on observed behaviour we avoid what I think of as "Please the listener" or "Please the interviewer" bias, whereas straightforward marketing questions could fall into this trap.

In order for me to make this point as helpfully and authoritatively as possible, can someone tell me what this bias is called, and if there is any research to demonstrate its magnitude?

Improve this question
1
  • This is a great question - but it is like opening Pandora's box :) Ill try and find a URL or 2 that you might find useful.
    – Nathan-W
    Commented May 25, 2010 at 11:11

6 Answers 6

Reset to default
16

I refer to it as social desirability response bias or acquiescence bias

Id start at http://en.wikipedia.org/wiki/Social_desirability_bias and explore from there to find some relevant research you could cite

You know I mentioned in my earlier comment - that this is like Pandora's box, and here's a page on wikipedia that supports that statement, just take a look at the number of different biases that can occur!

http://en.wikipedia.org/wiki/Category:Cognitive_biases

Improve this answer
2
  • 1
    "Social Desirability Bias" may well be the answer I'm looking for - thanks!
    – Francis Norton
    Commented May 25, 2010 at 11:43
  • 1
    I see this type of bias coming out in tests all the time. I always have to remind my staff that listening to a user's answers is not enough, you have to observe their behavior, the body language, the level of confidence they show using the mouse, the time it takes to make a decision... we've also encountered what my team calls "the village idiot syndrome" where otherwise intelligent users freeze up as soon as they are put into a test scenario...
    – Nathan-W
    Commented May 25, 2010 at 11:51
8

Stockholm Syndrome. I'm only half joking on this.

To fix the problem, I like to avoid asking any subjective questions like "Do you like that?" Rather, treat the users like Gorillas and you are Dian Fossey. Observe their behavior and make deductions.

User research is terrible for any sort of validation. You can't say, "They liked what we showed them so we must be on the right track." Rather use the sessions for discovery and exploration. I know its nearly impossible to convince executives to think this way. They want to validate so badly. It makes me so sad when executives do this. Its not helping build something awesome.

Improve this answer
3
  • I like this - I'm going to use that Dian Fossey line, in conversation if not actually in my report
    – Francis Norton
    Commented May 26, 2010 at 5:51
  • I'm Swedish and it happens to my user-test subjects all the time
    – JeroenEijkhof
    Commented May 28, 2010 at 1:01
  • 1
    Yes! Thank god someone said it! Validation is a terrible strategy. Users will say they 'like' almost anything put in front of them, because there's tremendous social pressures to do nothing else.
    – Jimmy Breck-McKye
    Commented Oct 10, 2011 at 0:17
5

Another term is Hawthorne Effect:

"[The] subjects improve or modify an aspect of their behavior being experimentally measured simply in response to the fact that they are being studied." (Wikipedia)

Improve this answer
1
  • +1 Yes, the 'Hawthorne' or halo effect is the term we learned in a module on my MSc HCI course. Participants perform in a way that they feel is desired by the moderator. In a nutshell it's people performing differently because they know they're under observation.
    – Janel
    Commented Oct 10, 2011 at 13:00
4

I've used the term coercive persuasion to describe a situation where the interviewer, and sometimes a group, strongly influences the ideology of a participant through cleverly disguised questions, such as "do you like this website?" or dropping in key words during the session discussion, and of course group peer pressure.

Improve this answer
1
  • 1
    Useful and interesting, but while I doubt that I will ever be able to fully cancel my own biases while user-testing, I'm more interested right now in the kind of bias that occurs from the participant's common-sense understanding of the test context, regardless of how carefully questions are phrased.
    – Francis Norton
    Commented May 25, 2010 at 11:40
4

Compliance.

http://en.wikipedia.org/wiki/Compliance_(psychology)

In psychology, compliance refers to the act of responding favorably to an explicit or implicit request offered by others

You also get 'Demand Characteristics' where users try to pre-guess what the correct outcome should be.

Improve this answer
0
1

Sounds like you're talking about Demand Charictaristics

In research, and particularly psychology, demand characteristics refers to an experimental artifact where participants form an interpretation of the experiment's purpose and unconsciously change their behavior accordingly

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.