My product team proposed a new feature for our mobile app and I'd like to get a sense of user interest for the feature.

I have the ability to run in-app surveys to our existing users but when running a survey of "would you want X feature?" I suspect there is a strong confirmation bias ("Of course I'd like an X!") that makes me suspicious of the survey results. What is a way to frame a feature survey differently or to analyze the results in a way that will mitigate this effect?

  • Users are notoriously bad at knowing what they actually need. Instead ask users about their experience in an attempt to find out which parts of your application need the most work. Also interview non-users to find out why they aren't using your application - this is where you can learn the most (meaning not from people who are already using your app). Great talk on the subject – Zach Saucier Sep 19 '15 at 15:13

Instead of surveying for a particular feature, ask about the pain point the feature should solve.

A great question would be something like "What is the worst thing about our app?" And then give three options.

Remember, users are crap about giving good suggestions about solutions, but nobody knows their problems more than they do.

You could always try asking your users to prioritize a list of 5 or 6 features - That way you're not asking "would you like to have X?" but "Which is more important to you U, V, W, X, Y, or Z?"

There's still a little of the confirmation bias there but you're also asking the user to trade off one feature for another and so reducing the effect of the bias.

  • 24
    By randomising the order of the choices for each participant, the notorious "first option is prefered" bias can be easily removed. – ojdo Sep 17 '15 at 17:59
  • @ojdo How does that help if each participant is more likely to choose the first option? You just spread the answers without gaining knowledge. Maybe rank top 3? – musiKk Sep 17 '15 at 19:19
  • 8
    @musiKk spreading the answers as ojdo suggests flattens out any anomalous answers (like first option bias) and allows the values from the less lazy participants to show through – Andrew Martin Sep 17 '15 at 19:29
  • Make the first option a nonsense option that's never going to be implemented anyway. – Mr Lister Sep 18 '15 at 16:27
  • @MrLister adding spurious options also has the potential to skew the result. How can you be sure that the false option isn't one that your users really DO want? – Andrew Martin Sep 18 '15 at 16:55

An approach I have used in surveys before is the open-ended question 'If you could make one change to the application, what would you do?'. This doesn't bias the user and has produced some really useful data about user's own concerns without unduly influencing them. For numbers of respondents up to a few hundred, the workload associated with analysing the data is reasonable. I tend to analyse by tagging and categorising the results.

  • 1
    So those are great but as you noted, once you have more than a few hundred responses the cost for analysis is too high. – SAR622 Sep 17 '15 at 14:50
  • Well, maybe. You can then start using text processing tools to get a general sense, or just divide the work up among more people. I – Peter Sep 17 '15 at 17:59
  • @SAR622 Wouldn't you first run a survey with just 100 respondents and then use a multi choice question in a second main survey (if you need more verification)? – David Mulder Sep 18 '15 at 19:19

Depending on the feature you could add a button for that feature (e.g. Share with a friend, if that's the new feature) in your navigation / interface and when people click the button you can show them a message that the feature is currently not implemented but soon will/might be. You can then gauge how many of your users actually click this button, and see this as a indication of how popular said feature will be. If the number of users clicking this button is below your pre-determined threshold, you can then decide not to build this feature.

Even better, you can then target these particular users that have clicked the button for more info about what they would expect from such a feature / what they were planning to do when they clicked the button. It's better to ask people about their current/past behaviour than ask them to predict their future behaviour.

  • 2
    Offering your users a button for a function that doesn't exist will severely damage the user's perception of the site/brand. A button that simply tells users that the function they've tried to access doesn't exist is just going to frustrate them. – Andrew Martin Sep 18 '15 at 21:52

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.