Does requiring NPS/satisfaction score to give feedback skew results?

Question

The other day I wanted to leave feedback on my gym's website. The feedback widget looks like this:

The first question is a mandatory satisfaction score (I clicked 10, but it's not the default).

I would have preferred to skip the satisfaction question because I wasn't in the mood to quantify my overall experience, I just wanted to send the feedback.

But, since it's mandatory - wouldn't users have a tendency to either give a very high rating or a very low rating to make sure their feedback is prioritized, regardless of whether they're "feeling the feelings", so to speak? And wouldn't that defeat the purpose of measuring satisfaction?

Answer 1

While this could differ across type of businesses* (putting a caveat on this generalization here), I have tested a similar consideration in the past by creating two different versions (randomized, as a sort of A/B test). One with the sat. item optional, and one with the sat. item as mandatory.

This did not result in a significant difference for me (and based on my sample size I believe the power was sufficient for this to be a valid in not rejecting the H0 hypothesis).

---

Given both versions had sat. results that came from 'surveys' piggybacking user initiated feedback (as I believe from your description is your example as well) it would be interesting to test these against standalone survey results initiated from the website/app side (unrequested by the user), mostly because this is a more conventional* sampling method for this type of sat. measurement.

* The organization I tested this for was not for a gym or sports business
* Easier to generate large sample sizes as you don't have to wait for a user to be pro-active, and perhaps fewer potential sampling biases by not selecting only those that have experienced a bug or other factors that can affect the impulse to give feedback