A well-known issue in "intercept" sampling is that it is not a probability sample of the customer population. If you were a grocery store interviewing 1/20 shoppers as they exit the store, for example, a customer who typically visits twice per week is twice as likely to participate in the survey as a customer who typically visits once per week.

Sometimes you don't care, because the transaction is the base you want to track. Grocery stores typically don't make any adjustments because, subject to certain simplifying assumptions, somebody who visits twice per week is exactly twice as important as somebody who visits once per week.

Sometimes you do care, because your image and how people talk about you is based on your entire population, not just your most frequent users. If you want the base for the analysis to be the individual user/customer, you would have to under-weight respondents by their frequency of visits or change your sampling procedure.

I'm trying to decide whether I should care or not. I'm updating a satisfaction survey for a website that currently has a certain percentage chance of appearing when the user logs out. Some users visit daily or more than daily; some visit less than once/week. More frequent users are more valuable, but it's not linear; a daily visitor is <7x as valuable as a weekly visitor.

If you were building a satisfaction survey for a website with regular users who vary in their frequency of use, would you base the analysis on the users or the visits?

2 Answers 2


I'm looking at your problem in a slightly different perspective. Please correct me if I'm wrong but I believe that you want to know if you should give any special importance to data collected from frequent visitors vs infrequent visitors.

A rating of 5 out of 5 by a frequent user may make your site great for frequent users but it may not make it great for an infrequent user. Instead of assigning importance to people's satisfaction survey based on their visits ask what makes a frequent visitor happy or unhappy and compare that to what makes an infrequent user happy or unhappy.

Typically the differences for software products/websites etc. will be: Infrequent user: looks for ease of learning to use the product and findability of things (ease of knowing what to do). Frequent user: looks for shortcuts, speed of going through their task when they know what to do.

Hope this helps.

  • Thanks Viraj, I think this is a really insightful approach... When you are having difficulty combining, stop combining!
    – Jonathan
    Sep 9, 2011 at 6:05
  • one of the issues is that we still often face the question of overall improvement. Did a change we make improve the customer experience overall. If both subgroups agree with each other, that's fairly definitive. If they disagree with each other, then perhaps we can try to apply a relative value judgment; but that's where it gets tricky.
    – Jonathan
    Sep 9, 2011 at 6:08
  • You are welcome. I'm glad I could help. :) When they disagree with each other it may be prudent to figure out whether one set likes a feature which the other dislikes or is it that your interface has inadvertently become great for only one set and the other set is not unhappy with the current features but wants more features that they think are relevant. Also things like showing in context help for newbies or the first few times a visitor comes along may help. This help may not be required later. Showing shortcuts to expert users may again help appease the frequent visitors/ expert users.
    – Viraj
    Sep 9, 2011 at 6:29
  • I'm tempted to declare this the answer, but I'll wait a day or two to see if anybody else has any ideas. :)
    – Jonathan
    Sep 9, 2011 at 18:04

The analysis will depend on what you want to get out of these data. As Viraj pointed out, there isn't necessarily one true “satisfaction level” but different user groups coming with different expectations and having a different experience.

You don't necessarily need to decide beforehand whether you should care or not. It's probably more interesting to collect some data to inform this decision, for example including a question about how frequently the respondents visit the site. That would allow you to do a couple of things:

  • See how much satisfaction (or any other outcome of your survey) depends on the self-reported frequency of use. If it doesn't make a big difference, you don't really need to worry and the rest of the discussion is moot.
  • Use frequency of use in other analyses: Weight observations to estimate average user satisfaction, include it as a moderator when testing changes, etc.

None of this is a black-and-white, all-or-nothing decision. For example, if you are looking at the effect of a change and you see it has a positive impact on satisfaction for unfrequent users but not for frequent users, you might want to compare the magnitude of the effect in both groups or present both the user and visit averages.

  • Thanks Gael! Frequency of use does make a big difference - daily users are a lot more satisfied than monthly users. Of course, that correlation probably has some causes and effects going in both directions.
    – Jonathan
    Sep 13, 2011 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.