My product team proposed a new feature for our mobile app and I'd like to get a sense of user interest for the feature.
I have the ability to run in-app surveys to our existing users but when running a survey of "would you want X feature?" I suspect there is a strong confirmation bias ("Of course I'd like an X!") that makes me suspicious of the survey results. What is a way to frame a feature survey differently or to analyze the results in a way that will mitigate this effect?
An approach I have used in surveys before is the open-ended question 'If you could make one change to the application, what would you do?'. This doesn't bias the user and has produced some really useful data about user's own concerns without unduly influencing them. For numbers of respondents up to a few hundred, the workload associated with analysing the data is reasonable. I tend to analyse by tagging and categorising the results.
Depending on the feature you could add a button for that feature (e.g. Share with a friend, if that's the new feature) in your navigation / interface and when people click the button you can show them a message that the feature is currently not implemented but soon will/might be. You can then gauge how many of your users actually click this button, and see this as a indication of how popular said feature will be. If the number of users clicking this button is below your pre-determined threshold, you can then decide not to build this feature.
Even better, you can then target these particular users that have clicked the button for more info about what they would expect from such a feature / what they were planning to do when they clicked the button. It's better to ask people about their current/past behaviour than ask them to predict their future behaviour.