Is asking 'balanced' leading questions the right way during user interviews?

Question

It sounds a little bit counter-intuitive, but I am wondering if we actually lose a lot of the information that we might otherwise intend on finding out or capturing by not asking more direct or 'leading' questions.

Conventional wisdom and the dogma of conducting interview would say that it is not a good idea, but this is because the general intent of asking these types of questions is to try and elicit a particular response, which in a courtroom or a marketing study might not be particularly appropriate.

However, in the context of a user interview trying to find out what makes people happy or unhappy, what their needs and wants are, wouldn't asking open-ended questions mean that you only end up finding out what they tell you instead of what you want to know?

My question is, can asking 'balanced leading questions', that is, if you ask them a question like "Is clicking on this button difficult for you to complete a task?" and then balancing it by asking "Is clicking on this button easy for you to complete a task?" the same or better than asking "What is the difficulty of clicking on this button to complete the task?"

Should these types of questions be included in user interview situations to gain better insights compared to simply asking open questions?

Answer 1

Asking closed questions is not optimal, for the reasons pointed out by Kristian - they skew the results. I think Adrian has a point too, since getting feature-specific feedback might be a better fit for a survey.

Moderation

It is the interviewer's job to moderate the discussion, to keep the focus. I think this is partly what you wanted/needed to accomplish, based on the quote below. This can be done with open ended follow-up questions. Or, indeed, sometimes with closed questions - see below.

Sometimes avoiding leading questions just because they are generally considered bad practice is counter-productive.

When to ask closed/leading questions

I think the right place for asking leading questions is to confirm gained knowledge.

You made an observation X during the interview. Based on that, you have induced a hypothesis Y.

Asking

"You said/did X. Is Y true?"

is fine, and to my understanding, the recommended way for proceeding with Contextual Inquiry (CI), for instance.

Interviews are limited

However, in the context of a user interview trying to find out what makes people happy or unhappy, what their needs and wants are, wouldn't asking open-ended questions mean that you only end up finding out what they tell you instead of what you want to know?

I think this is mostly true for interviews. An interview can only capture so much. That's why we have more advanced methodologies, like the CI, which include ethnographic methods for capturing the underlying user needs and wants.

In addition to asking the right questions, you need the context, the partnership, the interpretation and the focus principles the CI introduces. This doesn't mean interviews are useless - they just don't capture as much.

CI principles briefly

Context - observe the user in the real context

• DO: base your induction on concrete things - what the user actually does, instead what she says
• DONT: generalize or rely on abstractions.

Partnership - establish the master-apprentice relationship in order to reach a common understanding

• DO: set a common goal for the CI session in order to learn the master's needs
• DONT: leave questions unasked or break the egalitarian roles.

Interpretation - observe the master without bias and presumptions

• DO: observe what actually happens. Confirm your interpretation afterwards. (This does not bias the CI)
• DONT: make silent conclusions, lead the master with suggestions or with closed questions during the observation.

Focus - think critically, question your own hypotheses. Concentrate on the problem at hand that you're solving

• DO: challenge your own assumptions
• DONT: overlook conflicting actions or make assumptions.

Reference

H. Beyer, K. Holtzblatt, Contextual Design : Defining Customer-Centered Systems, Elsevier Science, Saint Louis, 1997, 497 p.

Answer 2

When you ask leading question you are biasing the responses because you "prime" the users to answer you in a particular direction. When you ask "Is clicking on this button difficult for you to complete a task?" it is more likely that the user will say that it is infact more difficult to click this button.

The point of asking not leading questions is to see in what direction the user will answer positive or negative. When you ask a leading question you are increasing the chance that the user will answer more towards the direction you've primed him.

I don't see any point of asking two "ballanced" questions because, you are wasting users time of asking 2 rather than 1 question (lowering response rate), and you are creating bias.

Lazar,J., Feng,J. and Hochheiser (2010) “Interviews and focus groups”, Research Methods in Human-­‐Computer Interaction, pp. 177-­‐212, Wiley.

Answer 3

One approach that can be useful for getting a meaningful response is to ask the user to place their response on a sliding scale between two options that have the same level of "goodness" or "badness", but different balance, then ask them if they think the change or idea is broadly good or bad.

The best way I can explain this is by showing how I'd do it in a survey and then suggesting a similar approach that could work in interviews.

If this was a survey rather than an interview, it could be presented like this:

Q1: How did you find this variation, compared to the current version? (circle one number)

1. Quicker to configure, but less fine control
2. -
3. -
4. -
5. -
6. More fine control, but slower to configure

Q2: Do you consider this change to be broadly positive or broadly negative (circle one answer)?

• Broadly Positive
• Broadly Negative

Neither end of the scale for Q1 is positive or negative in and of itself, but you can discern the effect of the change from it and decide if it's inline with the goal you were trying to achieve. You can then use Q2 to tell if this matches this respondent's needs - so you can test if your matching your goal AND if the goal is the right one.

Applying this to interviews

With some careful wording of multiple questions, you could use a similar technique in interviews - for example:

Q1.0: Is it more important to you that this control gives fine control of specific details, or is it more important that it be quick to use?

Q1.1: Where do you think the current approach falls on that range?

Q1.2: Where do you think this proposed change falls on that range?

Q1.3: Given your answer to Q1-Q3, do you think the change is broadly positive or broadly negative?

These wouldn't necessarily need to be posed as four separate questions - often, a verbose response to Q1 will give you answers to the rest of the questions. If you get a terse response to Q1, you could follow with the others to draw more information. That approach to questions is often useful if the interviewee is less inclined to talk beyond the immediate question.

Q1.1 asks the user to clarify their preference, and can be used to group the respondents.

Q1.2 sets a baseline for that respondent.

Q1.3 determines the difference to that baseline.

Q1.4 get a "good / bad" response based on the difference between Q2 and Q3.

Answer 4

I have seen this kind of question on here before.

This is the difference between Quantitative and Qualitative research.

High level information, such as what makes your users happy, is largely opinion based. For opinion based data to have any credibility you need to collect them in large quantities - A small sample may have their own biases and so may skew the result whereas, in a large sample the biases are smoothed. Although you can never eradicate potential biases completely (these could be regional, cultural, linguistic, etc.) the larger your sample, less likely they are to generate spurious or anomalous results.

On the other hand, collecting the detailed data required for more definitive research (like usability for instance), where the results are not so opinion based, Would be extremely difficult, time consuming, and expensive to do on a large scale.

This is why we tend to avoid asking questions about the users opinion during interviews - they're simply not worth asking on that scale.