2

I’m working with a stakeholder on a potential major pivot to a product, which gets used both inside and outside our organization. The stakeholder wants to directly ask a question like this in a survey to internal potential users: “If our organization offered a way to (achieve your primary goal) (without either of your two top pain points), would you be interested in using it?”

I see this question as biased because it looks like “who in their right mind wouldn’t say yes?” The stakeholder has worked for us for many years and insists that this won't be a problem, but I find this hard to believe.

Also, in our context, we have to consider whether or not our users have the time, capacity, and desire to use a product like ours. They can use it in or outside their jobs, but they don’t have to.

Are we out of line to ask a question like this? Even if we are, what is a less biased way to ask it?

2

From the way you've put parts of the question in parentheses, I interpret that the actual proposed question reads something like:

"If our organization offered a way to book travel without having to check other websites and with a cheapest-price guarantee, would you be interested in using it?”

with actual, specific information like that in both of those fields. In this case, it's understandable why the stakeholder would consider this question reasonable. It's exactly what they want to know! If we implement this feature, would people use it?

However, your intuition is correct. The answers you get to this question will not provide the insight your stakeholder desires. What the stakeholder seeks to find out is "Should we implement this feature or not?" and the metric they are trying to map to that question is "What kind of adoption rate increase would we see if this feature were implemented?" Unfortunately, the investigation they want to perform is "Will users claim they would increase adoption rate if we implemented the feature in question in whatever way that user presumes we would implement it?"

@omkar-chogale has the right of it. Seek to understand the needs of the users. If your stakeholder is just seeking confirmation that some feature which has already been identified through the usual ways would actually be really valuable to users or would actually result in some meaningful increase in adoption, maybe a pilot study would yield useful evidence. Show some focus group the proposed feature as a mockup and ask for feedback. Better, show them a few possible new features as mockups and get rankings and ratings for each.

  • 1
    +1 Thanks for your contribution to UXSE. You might be a new contributor but it looks like you've dealt with your fair share of UX issues in your career. Looking forward to seeing some more of your answers :) – Michael Lai Aug 22 at 4:44
  • Thanks, Michael! I found the place through the "out of beta" announcement and thought it would be a nice forum to learn and share. – sintax Aug 22 at 16:04
1

You are right. This really needs a deep understanding of context.

However, do not tell users what are their pain points ( this creates a bias). Instead, ask them what are their pain points, and goals. Survey is to get their views not to confirm your beliefs.

Also, ask them what is the motivation for them to use existing system. See if that is accomplished in your proposed system, otherwise user's decision making might be difficult.

1

That's definitely a tricky situation to be in. If you are conducting some qualitative user research you could possibly use the data to confirm the pain point and support the goal which you are working towards.

Another way I can think of is: If you want to add it as a survey question it might be good to ask:

How useful would this feature/functionality be for you on a scale of 1-10. Maybe that could be used to assess if the user finds that feature/item useful. Also additional you could ask a follow up question as to why did you select that score to obtain more information on why the suggested feature might be a good/bad fit for them.

1

It seems to me that you'll want to present this through two multiple-choice questions, both short and to the point and with a limited range of possible answers, but more than a single presumed goal (data point 1) and more than two top-tier, also presumed, pain points (data point 2). To that end, you'll need info on alternate goals and pain points - hard to get into more detail without domain information (which is likely classified, I quite understand).

In order to set this up, perhaps you could quiz your stakeholder, casually and innocently of course, on goals or pain points they previously assumed their users had, and how they found out that these assumptions had no base in reality and hence need no further investigation. That will give your interlocutor an opportunity to shine with the sheer brilliance, depth, and acuity of their insights (and more to the vanity piece below).

You may expand that pain point piece into querying the nature of participants' pain point; financial, time, or stress and aggravation. Ask for a response ranking here, or else people say 'all of the above and then some' and you get shoddy data.

For valuation insight, this gets tricky. A slightly gamified research method comes to my mind; it's called Buy-me-a-Feature. By this method you place a dollar value, or some equivalent monetary expression of material worth, next to a given characteristic of your offering. It's a bit like App Store honesty: I will always gravitate toward an app that's $9.99 with a concise description of its feature set (good ratings are an added bonus), over some vaguely stated value proposition with 'in-App Purchases'.

Buy-me-a-Feature unfortunately requires deeper marketing insight or else it can quickly become hokey and cause loss of credibility unless you can really gamify the experience and use Monopoly money. And when used as a survey rather than in a face-to-face research situation it presumes that something is actually for sale. Neither may apply to your situation.

No offence but it sounds like your stakeholder has a severe case of salesmanship bias, and surveys to such a tune do little but annoy consumers. Plus the age-old trite statement that "we know our customers" does little but annoy designers!

You may need to play to that person's manifest vanity a little, and if that isn't going anywhere I would have hard business ethics take over and state rather bluntly that a mere confirmation survey, such as unfortunately pushed on you, will just not present enough value to justify the expense - and frankly that time is better invested in clean UI detailing to at least make a poorly researched offering present in a shiny way. If you're in a consulting situation - even in-house - that may be the Ultima Ratio. After all, you're the professional expert whose advice your stakeholder seeks, and he or she must be treated with honesty. If your patient insists on the personal benefits of smoking, what are you as their doctor going to be telling them?

Sometimes anecdote telling does the trick:

Presumed goals and pain points are a little like these telemarketing calls we get for furnace cleaning (I live on the 18th floor of a high-rise condo building, alas no fireplace), dog grooming (we have new-ish furniture, allergies, and a love of travel, hence no poochies even though we love 'em) or that extra credit card I must have (no thanks, I want to make more money, not owe more). Even if one of these things ever becomes a true need, I will most certainly not enlist the telemarketers' services to satisfy them. It's just not good business.

To sum up: If you manage to convert your survey into a slightly expanded multiple-choice questionnaire with drilldown on pain point categories, you might gain a lot.

Difficult situation, to be sure - best of luck!

1

Theoretically there is always going to be some degree of bias in research, so it would be more appropriate to aim for practices that aim to minimize the degree of bias.

The problem with the way the feedback is being collected isn't so much to do with the question being a little bit 'loaded' so that you are likely to get a particular response (but you should be concerned about people who don't have issues with this approach to gathering feedback). I see the main problem as not being able to independently verify the connection between what the actual user pain points are and whether your products and services does indeed address those pain points.

So I see the bias as a minor issue compared to the fact that you won't actually gain any useful information. However, if they insist on asking this particular question, it is important to add additional questions to validate what the actual pain points are, and whether your products and services do address these pain points.

As you say, in your context you also have to consider whether or not your users have the time, capacity, and desire to use a product like ours. The way the question was asked doesn't necessarily take those context into account, so again it is difficult to know what you can do with the responses if you are going to ask questions the way it has been proposed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.