I've read that it's best practice to calculate a sampling interval (eg the survey will appear to every fifth or tenth visitor), based on your traffic, how long the survey will run, and how many responses you need. But I don't understand why.
Wouldn't it be better to get as many responses as possible? As long as the survey runs over a couple of weeks, so you don't get any bias from the time of the responses, how would this restriction improve the validity or reliability of the data?
Why would you annoy all of your users with a survey when one tenth is sufficient to give a statistically significant result?
Given sufficiently high traffic, you'll be able to get n responses in as short a time as you like.
But, that sample is likely to be biased in time and/or geography, especially if you get them within an hour or so.
If you only collect responses between 11am and 1pm on a Monday, London time, then you'll be catching a bunch of people from the UK who check out your site at lunchtime, but almost noone at all from New Zealand or Australia.
Collect responses between 5pm and 9pm on a Friday night, New Zealand time, you'll miss most everyone in the US (too late at night), catch the after breakfast crowd from the UK and you'll miss all the locals out on the town.
Introducing a sample interval to spread capture out over several days allows respondents from different timezones and different locations to all respond.
Abstract: surveys are annoying your users, and sample bias is worse than sample error.
A large sample size isn't as important as an unbiased sample selection.
Standard deviation decreases with 1/sqrt(N), so doubling sample size will decrease the sampling error by roughly 30%.
So if you need to bother 1000 people to figure out 76% +/- 6% want or are OK with more cowbell, doubling the sample size may bring you to 78% +/- 4%.
That's not as much of a gain in information. With a well selected sample, surprisingly few candidates are required for good results. However, if a segment is underrepresented, your results can be useless.
In above example, imagine only a group of New Zealanders is "strongly opposed" to more cowbell. They are your best customers actually, but they log on to your site only every second saturday night when "tractors and tyres" is not on TV.
Yes, it would be better to get as many responses as possible. However, when writing a research plan you would calculate the amount of responses that you need. When you combine this number with the amount of visitors and keep in mind that maybe 5 to 10% might be engaged enough to fill out the form, you can come to the conclusion that not everybody needs to see this.
Also, not all pages may need reviewing. A certain page scope or user flow may apply.
And besides that, I think these types of research are mostly initiated by the marketing department. Who are mostly very quantity-loving.
But the fact that the current method of displaying these surveys tends to be annoying is, I think, one of the main aspects of not showing it to everyone. Hopefully things will shift more and more towards things like Userecho, where the user himself decides whether he has something to complain or praise on the website.
Unless you are managing the application of the survey itself, you will be paying for the survey company's overhead and for the sample you are hoping to achieve. If you have adequate details about your users (sex/age/income), you can tailor your sample and target the minimum number of users until you assemble an effective sample.
Minimizing exposure to your survey:
Pushing a survey at all distracts your end user, can compromise your brand (adds at least the complexity of "if I go to that site, I might get a survey thrown in my face"), and lends volatility to the experience overall; minimizing exposure to that possible variable in the users' experiences preserves the coherence of whatever narrative your users have in mind for when they visit your site.
A tertiary concern; surveys, particularly ones where you are looking for preferences and feedback about your product, can be a double-edged sword for the participant. They may feel some satisfaction at having had their concerns and opinions valued and considered. They may feel that their answers are not valued, however, if nothing changes or they feel there is no direct connection between investing their opinion and the site. Over-saturation of survey prompts may embattle that disenfranchisement, and distribute it among your users. Unused opinions have a way of inculcating a perception of a brand and environment. If you provide an ineffective way for leveraging those opinions, and fail to deliver, it erodes confidence.
