User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
I have been using a sample size calculator to know how many participants I should have to have a sample and results that are representative for my total population. But what if we don't know the total population? how do you calculate your sample?
Thanks in advance,
About 400 people. Sample size is more influenced by your confidence level and margin of error than the population size.
If your confidence level is 95% and your margin for error ±5% your sample size is 370 for a population of 10,000 or 384 for a population of 1,000,000. But from 1 million on up to 10 billion, the sample size only increases by 1.
But for a population of a reasonably large size (say 1 million), your sample size needs to be between 68 (90% confidence, 10% margin) and 16,307 (99% confidence, 1% margin).
Some calculators (like this one) will let you omit the population size.
You can also go for the largest sample size you can get, and then work backwards to see how confident you can be in the results. Maybe you can only get 100 responses, but if they all agree, that might be enough.
Good answer by Nathan and 400 is a safe sample size in survey-based social science studies. Another aspect to consider is the type of analysis and tools used for those analyses. For example, if the analysis involves exploratory factor analysis (EFA) using SPSS and confirmatory factor analysis (CFA) using AMOS, a minimum sample size of around 300 is the minimum, and a larger sample size would result in better estimates/loadings (Netemeyer et al. 2003, p.116).
If the estimates involve partial lease square (PLS) based analysis, such as using SmartPLS, a much lower sample size would work; see this forum post about '10 times rule'. For example, if a study involves survey data from marketing professors at top universities in Australia, the sample size would be very small for practical reasons. In that case, a method/tool that produces reliable estimates with lower sample size, such as SmartPLS, needs to be considered.
A study can involve different types of analysis using multiple data sets, and the sample size would vary. For example, if the study involves multi-group analysis, each group needs to have a sufficient sample size. A good example of different sample sizes for different studies in the same research project is available in this article.
Overall, the types of analysis necessary for a study, the nature/availability of the respondents, and the expected/required confidence level should be considered when setting the sample size.
There are two sides to this discussion -
sample size. This is something that Nathan and Syed have covered adequately.
sample selection. This is something that's actually more important.
Suppose you are doing this for a consumer product. Different possible user groups might be college kids, employed people, men with kids, women with kids, retirees, etc., etc. You need to work this out based on the product in question.
These groups vary wildly in their purchasing power, interest level in different types of products/services, etc.
If you talk to 400 college kids only (low purchasing power, enough spare time, highly sociable, interested in fads and novelties), you might get a very very skewed perspective. And if this research were for a home mortgage product, your results would be summarily meaningless.
It is more beneficial to talk to 10 highly targeted/relevant users than to talk to 1000 random people - unless your subject of research is something extremely generic.
For many highly targeted groups, you probably will find it extremely hard to get to 400 users to talk to you. Sometimes these many users might not even exist. E.g. investment bank directors, governors of provinces, etc. If you are talking about a software product that solves a specific problem a group like this has, it might suffice to have meaningful conversations with 10-20 people in this group.
