# Why would you only show a survey to one in (n) visitors?

I've read that it's best practice to calculate a sampling interval (eg the survey will appear to every fifth or tenth visitor), based on your traffic, how long the survey will run, and how many responses you need. But I don't understand why.

Wouldn't it be better to get as many responses as possible? As long as the survey runs over a couple of weeks, so you don't get any bias from the time of the responses, how would this restriction improve the validity or reliability of the data?

-

Why would you annoy all of your users with a survey when one tenth is sufficient to give a statistically significant result?

-
Additionally, if your website doesn't require logging in, you're less likely to hit the same person twice the larger a value of "n" you use. – aslum Aug 21 '12 at 16:39
Not really true. You're still annoying the same number of users all the same, because you have to run the survey longer for the same sample size. – Jimmy Breck-McKye Aug 21 '12 at 20:51
That's not what he's asking .. "Wouldn't it be better to get as many responses as possible?" – Erics Aug 21 '12 at 23:09

Given sufficiently high traffic, you'll be able to get n responses in as short a time as you like.

But, that sample is likely to be biased in time and/or geography, especially if you get them within an hour or so.

• If you only collect responses between 11am and 1pm on a Monday, London time, then you'll be catching a bunch of people from the UK who check out your site at lunchtime, but almost noone at all from New Zealand or Australia.

• Collect responses between 5pm and 9pm on a Friday night, New Zealand time, you'll miss most everyone in the US (too late at night), catch the after breakfast crowd from the UK and you'll miss all the locals out on the town.

Introducing a sample interval to spread capture out over several days allows respondents from different timezones and different locations to all respond.

-

Abstract: surveys are annoying your users, and sample bias is worse than sample error.

A large sample size isn't as important as an unbiased sample selection.

Standard deviation decreases with 1/sqrt(N), so doubling sample size will decrease the sampling error by roughly 30%.

So if you need to bother 1000 people to figure out 76% +/- 6% want or are OK with more cowbell, doubling the sample size may bring you to 78% +/- 4%.

That's not as much of a gain in information. With a well selected sample, surprisingly few candidates are required for good results. However, if a segment is underrepresented, your results can be useless.

In above example, imagine only a group of New Zealanders is "strongly opposed" to more cowbell. They are your best customers actually, but they log on to your site only every second saturday night when "tractors and tyres" is not on TV.

-
+1 this answer excellently supported & well articulated! – Valeria Spirovski Aug 23 '12 at 2:41

Yes, it would be better to get as many responses as possible. However, when writing a research plan you would calculate the amount of responses that you need. When you combine this number with the amount of visitors and keep in mind that maybe 5 to 10% might be engaged enough to fill out the form, you can come to the conclusion that not everybody needs to see this.

Also, not all pages may need reviewing. A certain page scope or user flow may apply.

And besides that, I think these types of research are mostly initiated by the marketing department. Who are mostly very quantity-loving.

But the fact that the current method of displaying these surveys tends to be annoying is, I think, one of the main aspects of not showing it to everyone. Hopefully things will shift more and more towards things like Userecho, where the user himself decides whether he has something to complain or praise on the website.

-

Unless you are managing the application of the survey itself, you will be paying for the survey company's overhead and for the sample you are hoping to achieve. If you have adequate details about your users (sex/age/income), you can tailor your sample and target the minimum number of users until you assemble an effective sample.