When to adjust for frequency of use in satisfaction surveys that use intercept sampling?

Question

A well-known issue in "intercept" sampling is that it is not a probability sample of the customer population. If you were a grocery store interviewing 1/20 shoppers as they exit the store, for example, a customer who typically visits twice per week is twice as likely to participate in the survey as a customer who typically visits once per week.

Sometimes you don't care, because the transaction is the base you want to track. Grocery stores typically don't make any adjustments because, subject to certain simplifying assumptions, somebody who visits twice per week is exactly twice as important as somebody who visits once per week.

Sometimes you do care, because your image and how people talk about you is based on your entire population, not just your most frequent users. If you want the base for the analysis to be the individual user/customer, you would have to under-weight respondents by their frequency of visits or change your sampling procedure.

I'm trying to decide whether I should care or not. I'm updating a satisfaction survey for a website that currently has a certain percentage chance of appearing when the user logs out. Some users visit daily or more than daily; some visit less than once/week. More frequent users are more valuable, but it's not linear; a daily visitor is <7x as valuable as a weekly visitor.

If you were building a satisfaction survey for a website with regular users who vary in their frequency of use, would you base the analysis on the users or the visits?

Answer 1

I'm looking at your problem in a slightly different perspective. Please correct me if I'm wrong but I believe that you want to know if you should give any special importance to data collected from frequent visitors vs infrequent visitors.

A rating of 5 out of 5 by a frequent user may make your site great for frequent users but it may not make it great for an infrequent user. Instead of assigning importance to people's satisfaction survey based on their visits ask what makes a frequent visitor happy or unhappy and compare that to what makes an infrequent user happy or unhappy.

Typically the differences for software products/websites etc. will be: Infrequent user: looks for ease of learning to use the product and findability of things (ease of knowing what to do). Frequent user: looks for shortcuts, speed of going through their task when they know what to do.

Hope this helps.

Answer 2

The analysis will depend on what you want to get out of these data. As Viraj pointed out, there isn't necessarily one true “satisfaction level” but different user groups coming with different expectations and having a different experience.

You don't necessarily need to decide beforehand whether you should care or not. It's probably more interesting to collect some data to inform this decision, for example including a question about how frequently the respondents visit the site. That would allow you to do a couple of things:

• See how much satisfaction (or any other outcome of your survey) depends on the self-reported frequency of use. If it doesn't make a big difference, you don't really need to worry and the rest of the discussion is moot.
• Use frequency of use in other analyses: Weight observations to estimate average user satisfaction, include it as a moderator when testing changes, etc.

None of this is a black-and-white, all-or-nothing decision. For example, if you are looking at the effect of a change and you see it has a positive impact on satisfaction for unfrequent users but not for frequent users, you might want to compare the magnitude of the effect in both groups or present both the user and visit averages.