As a customer, I get sent a decent amount of surveys, particularly from larger companies.
A lot of the time they ask for 5 or 10 minutes of my time to fill out feedback. Typically I ignore these as there is no direct benefit to me to fill it out.
Sometimes they offer an entry into a prize draw, or for more in-depth questions will pay a sum of money or gift card.
It's safe to generalize and suggest that both the positive and negative effects of incentive use on survey response rate and data quality can be described as mild to none.
I first included a longer list of links, but on review found their actual content to be thin, or they had based their content on other studies referenced in the list.
The first two studies (Impact study by International Journal of Market Research, and the Meta Analysis by Singer, 2012) could be considered core works in this area and are frequently referenced by other works.
Experiment 1 was carried out in a commercial market research panel. It examined whether three different types of promised incentives (redeemable bonus points, money lottery and gift lottery), four different amounts of bonus points or raffled money, and two different denominations of raffled money influenced response quantity, sample composition, response quality and survey outcome. Type of incentive and number of bonus points mildly influenced dropout and sample composition. Moreover, response was higher with bonus points than with the two types of lotteries. Response quality and survey outcome were not affected.
Experiment 2 was conducted in a non-profit panel, which holds one half self-selected and one half non-self-selected participants. Incentives were two different amounts of raffled money in two different denominations. Response, dropout, response quality, survey outcome and sample composition were not affected.
Response Rate
Church’s meta analysis (1993) found:
(Cross-sectional Mail Surveys)
- Prepaid incentives yielded significantly higher response rates than promised or no incentives
- Monetary incentives yielded higher response rates than gifts
- Response rates increased with increasing amounts of money, though not necessarily linearly
- Edwards et al. (2002) reported similar results in a subsequent meta analysis
- With very few exceptions, reports of more recent experiments are consistent with results reported by Church and Edwards et al.
(Interviewer-Mediated Surveys)
- Singer et al.’s meta analysis (1999) of 39 experiments found results similar to those in mail surveys, though effects of incentives were generally smaller
- Cantor, O’Hare, and O’Connor’s analysis (2008) of 23 RDD experiments found:
- Prepayment of $1 to $5 increased response rates from 2-12 percentage points over no incentives
- Larger incentives led to higher response rates, but at a decreasing rate
- Effect of incentives has not declined over time, but baseline RR dropped substantially
- Prepaid incentives at refusal conversion had about the same effect as those sent at initial contact, but at a lower cost
- Promised incentives of $5 and $25 did not increase RR; larger incentives sometimes did
- These findings are generally consistent with other experiments involving interviewer-mediated surveys, including face-to-face surveys
Response Quality
(Response quality most often measured by item nonresponse and length of answers to open-ended questions; other measures would be desirable (e.g. accuracy, reliability))
Two alternative hypotheses about such effects
Singer and Kulka (2002) found no support for decline in quality, and modest support for alternative
Since then, the small number of studies (mail, RDD, and face-toface) that have examined incentive effects on data quality have, with one exception, found no effects. The exception is Jäckle and Lynn (2008), who found incentives increased item nonresponse in a panel study but decreased unit nonresponse, resulting in a net gain of information
Cantor, O’Hare, and O’Connor (2008) argue that the two hypotheses need to be tested controlling for factors such as survey topic, size and type of incentive (e.g. prepaid, promised, refusal conversion), and whether studies are cross-sectional or longitudinal. For this, a much larger pool of studies would be required
Medway (2012) has recently examined this question using a much larger pool of measures of effort (e.g. item nonresponse, length of open-ended responses, straightlining, interview length, underreporting to filter questions, lack of attention to question wording, use of round numbers, order effects, etc.) as well as the potential interaction of a number of demographic characteristics with receipt of an incentive
Vehicle: Experiment embedded in JPSM Practicum survey; N=1700; RR~16%: half received $5 prepaid incentive, half none
Results: - RR: 22% with incentive, 11% without - Cost to complete/case: $57.68 with incentive, $63.76 without - Significant differences on only 2 effort indicators—reduced item nonresponse and less time to complete; neither was significant once cognitive ability and conscientiousness were controlled - No significant interaction effects between demographics and incentives on an index of satisficing
Overall, this study, using hundreds of thousands of student respondents from over 600 colleges and universities, found little evidence that survey incentives negatively affect data quality. Our analyses showed minimal differences between incentive and non-incentive groups with regard to straight-lining, item skipping, total missing items, and completion. Contradicting Barge and Gehlbach’s finding, we found, in fact, that incentive respondents actually had better data quality than non-incentive respondents. Though the effects were small, they consistently favored the incentive student group.
So incentivizing surveys can increase response rates (great!)–but may encourage satisficing (not great). In order to cut down on people who only take your survey just to earn a prize, you may want to try using indirect rewards. In fact, we know all about indirect rewards [...] instead of paying our survey respondents to take our surveys, we donate 50 cents to the charity of their choice every time they fill one out. That way, they’re less likely to rush through surveys out of pure self-interest.
Yes, surveys and research are affected, but it is unclear how much.
Incentives that can be redeemed for straight financial benefit can and will attract people that are in it for the reward.
Barge & Gehlbach (2012) reported that respondents receiving a $15 incentive were much more likely to satisfice as indicated by increased item skipping, rushing (shorter duration), and straight-lining compared to those that received no incentive. The authors also reported that this increase in satisficing was also associated with a decrease in data quality and, more specifically, scale reliability.
But,
However, this study did not parse out the effects due to incentives. Thus, the direct impact of incentives on scale properties and parameter estimates is still not well understood.
Source: this study conducted in 2015.
The problem is that it's very difficult to conclude if this behaviour only occurs because of the incentive, or if the same people would display this same behaviour if there were no incentives. Since using incentives does highly increase the amount of people willing to fill in the survey, it is a matter of weighing risk and reward. Is it worth it to slightly alter the results, if it means there will be more response? If the answer is yes, then by all means use incentives. If truthfull reporting is absolutely necessary, don't use incentives and attract only those willing to do it out of sheer good will.
@dennislees has a lot more studies listed that are worth looking into.
Feedback is one of the most important ingredients for any company.
Why?
Because everyone wants to reassure that they are moving in the right direction. I was reading a book regarding lean startup where the writer invited many guests to physically visit their office and tried to influence them in using the product as well as to share it with their friends. Merely asking the invited guest to share the product helped the writer to understand the reaction of that person which matched with many others who were invited in the same way.
Feedbacks provide surviving or guiding insights for the company, but what is in for the invited people? they have spent time and maybe some money for coming in and even if don't like your product, it still counts as an insight.
Adding rewards for providing honest opinions attract users as they feel that they are doing something productive rather than just advising for the better good.
But I see a lot of companies try to trick people who are willing to provide feedback for some rewards, leading to less number of audience who are willing to spend precious time to provide feedback even if there are rewards associated with it.
You are cutting your own legs if you are tricking people for feedbacks.
Best strategy would be "Don't promise anything" or promise a decent reward.
For example, you are the first person to manufacture fidget spinner and want to know if people will play with it or will throw it away.
Starting with prime target audience with age group (12-18)yrs
Reward - 3 Fidget Spinners.
A little beyond your target audience age group ( 34-30) yrs
Reward - 3 fidget spinners and 2 movie tickets (with popcorns).
Hence, merely adding some exciting rewards won't cut it. Promising i-phone to one of 100 people will get you 99 angry guests at the end but for the same cost if you are able to provide 100 people with your product and little but useful incentive then hallelujah you have 100 happy guests willing for more feedbacks in future. In addition, 100 people have your product which they will share with their relatives and friends.
It totally depends on how you are strategizing your surveys to get more benefits out of them.
Adding rewards with the survey will definitely get you more attraction but to build upon it, you need to think what the rewards should be.
