3

So I'm new to research and I started a survey to understand users a bit more. After doing a HotJar Poll/ Email Survey on one of the sites I'm managing I'm getting 55+ as an age group for the majority of users who responded. Both Google Analytics & Alexa Pro are reporting the majority of users in the 18-24 range. What can be done usually about that?

Also if some of the respondents skipped some questions. Do I disqualify them?

Thanks

  • 1
    This is interesting - I wonder if either some of the users reported by your analytics have misrepresented their age or the survey attracted more 55+ users and was simply dismissed by the younger users. With regard to skipped questions, ideally your survey will have been designed so that each question can stand on it's own so that you can analyse the answers without having to refer to too many other data points for context. This means that you shouldn't have to disqualify a respondents entire answers just because they skipped some questions - the answers they did provide are still valid. – Andrew Martin Aug 23 '18 at 8:45
  • Makes sense. I'm leaning towards being dismissed by younger users but was wondering if this is a common issue. Thanks a lot. – Raiden Aug 23 '18 at 9:03
  • 1
    Is either (a) the 55+ selector pre-selected (so someone not interacting with that question be thought of as 55+), or (b) are users allowed to not make a choice about their age, but if they do so, will it fall into the 55+ category by default? – TripeHound Aug 23 '18 at 10:54
  • 1
    I'd step back and ask yourself the question: "why, as a user, should I spend my time filling out your survey" What does the user get out of it ? ( and if you understand this then this may be what's skewing your response rates ) – PhillipW Aug 25 '18 at 10:27
  • Hi @Raiden, thanks for contributing to our pool of knowledge at UXSE! What do you mean by what can be done about that? I think you should clarify the purpose of the research because that would allow us to consider whether age is going to be a major factor in analysing the results. If it is purely about behaviour and age doesn't affect the behaviour (even though it is likely) then you probably don't have to do too much about it except to keep it in mind. – Michael Lai Aug 28 '18 at 23:35
2

I think it could be multiple things:

  • If the users are self-reporting their ages, they might be not comfortable with sharing that information. If this covariate is not relevant to your analysis, maybe try adding it as an optional field. You can try and get this data by asking other questions that aren't so direct and would still place them in a particular age category. But if you really don't need this covariate, I'd just leave it out and assume your sample will be random enough for this experiment to work.
  • It might also be the case that not all visitors are interested in your survey, so you are getting more respondents from a specific age range. Is there any other source of data you can use to determine how big each group is, and figure out better ways to solve your sampling issue? Did you, for example, get a different distribution on your email responses vs. website?
  • About skipping, I'd actually worry about survey noncompliance if the skipped questions are generally the same or if a particular group is jumping questions at a higher rate. You might end up getting data only on people who feel good or positive towards your survey and disregarding the rest.

UPDATE: Validating the Experiment

I'd start validating the sample first, to make sure that the discrepancy between the expected and observed exists. If analytics is reporting visitors in the 18-24 range, and they are interacting with the site in some way, there are probably records of their interactions like emails, social accounts, etc. Normally I'd expect the distribution of users to be similar in all channels. If not, the problem is presumably related to the measurement tool, in this case, Google Analytics.

Let's say the measurement tool works fine and we do indeed have the reported distribution, then we need to verify if the survey prompt was equally distributed. If the plan was to assign the survey randomly, were the requirements met? Are we observing conditions which could make a certain group of users be active during a certain period of time, a moment of the week etc?

If this is not the case, then we need to look at pre-existing conditions that could have influenced their responses, maybe the majority of non-respondents participated in a previous experiment that was run on the site and, in the process, their confidence when interacting with the site was affected.

If we find no evidence for this, I think the simple solution is to redesign the survey or try other survey methods like interviews.

  • 1
    +1 you have listed some of the behaviours that can bias the results from the surveys and that's great! How would you normally go about validating the information just to complete the answer :) – Michael Lai Aug 28 '18 at 23:36
  • @MichaelLai Thanks, I think the validation can take multiple steps. I've added a brief explanation on things I'd look at if I had a similar situation. – ghislaineguerin Aug 29 '18 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.