1

I am gearing up for my first ethnographic field study of two to three different groups of users who use the same system. I'm trying to figure out if they use it in different ways even though their goal is the same. I have been searching around and haven't found a good source on the number of people I should speak with in each group. I'm looking for the minimum amount. I'm going to be hard pressed to get more than two or three in each group but if I know I need more I'll get them. For usability testing I test six people, but I can't decide if I should test six in each group, six total( since their goals are so similar) or something else entirely.

NNg indicates that its qualitative research so I don't think I need a huge group of people: http://www.nngroup.com/articles/which-ux-research-methods/

This articles tells me to cast a wide net to recruit but still doesn't mention what the end goal is: http://uxmag.com/articles/making-the-most-of-ethnographic-research

There are various articles that talk about how to do the study but still nothing about how many people to actually study. Any articles or resources you can direct me to would be appreciated as well.

4

Keep collecting data until you get to the point where you stop hearing new things.

I'd also look at how Ethnography is done in Anthropology as this is where the approach actually comes from:

Eg: http://www.discoveranthropology.org.uk/about-anthropology/fieldwork/ethnography.html

  • 1
    This. In the academic literature this is often referred to as "saturation". Googling around that should get the OP some pointers. – adrianh Feb 19 '15 at 13:52
  • Well, that's going to be an interesting sell to the people who like solid numbers, but I'll work on it. UX (unfortunately) is often a hard sell anyway. Thanks! – Fletchling Feb 19 '15 at 22:45
3

TLDR: Sample size required is highly dependent on what you're trying to measure and study conditions. In an ideal environment, you need 6 per group for small number of groups for statistical significance. In a more typical study, you probably want 10-12 per group. If you don't care as much about statistical significance, by all means use less.


How do you determine sample size?

There's 3 main factors that affects sample size for obtaining statistically relevant results when you're making comparisons between groups.

  • Type of data you're collecting and the size of the difference between the groups you're trying to observe. This affects: statistical power, your ability to detect a difference should it exists between groups. The more precise the measuring method and the larger the overall difference between the group in relation to the preciseness of your measuring method, the smaller the sample size you'll need per group. Some tests are also inherently better at detecting differences. E.g. if you're testing to see if X is larger than Y, but don't care if X is smaller than Y (1-tailed test), then your statistical power is twice as large as if you want to see if X is different, larger or smaller, than Y (2-tailed test). Data may also be continuous (e.g. intensity of color) or discrete (e.g. Is either Red or White). Tests for continuous distribution are inherently more powerful than discrete tests and requires smaller sample sizes.
  • How confident you must be that the difference you observe truly exists and it didn't show up by chance. Sometimes people refer to this in the reverse as "margin of error". You'll need larger sample size per group if you need more confidence.
  • How much variability is there in what you're trying to measure. The more "noise" you have compared to "signal strength", the larger the sample size you need. Behavioural studies tends to have way of variability as compared to physical studies with objects.

But I'm sure what you're hoping for are actual numbers...

So here's an example of a properly controlled, and randomized study in a lab.

If you are talking about observing a difference in 1 variable statistically (T-test) for a medium (~20-30%) difference in a controlled behavioural experiment, you need 6 in each sample to observe this difference 80% of the time. The numbers required for an analysis of variance (ANOVA) for 3 or 4 groups are around the same as well. If you have more groups, or the difference you're trying to observe is smaller, you'll need more per group. I don't have the statistics textbooks in front of me to quote stuff. But this is commonly used for pilot behavioural studies in drug research.

When you deal with field studies, you'll have way more variability. I don't know how much more in your case, but a typical power vs sample size tends to be an exponential curve. There's a point in which you get very little extra value per additional tester you add.

Power vs Sample Size Curve

So a ballpark "reasonable" number (I'm pulling this out of my ass for a typical semi-controlled study where you need for statistical significance results) is 10-12 per group. Above 20, you're usually not getting much bang for your buck for most readily observable difference between groups.

Also, groups of unequal sizes decrease sensitive of tests. i.e. a test with groups of 6 & 6 have much higher statistical power as groups with 5 & 7.

There's an excellent article on Measuring U on how to determine your sample size. Note: their example assumes they're looking at survey results with a completion rate of only 50-70%. That's why their sample sizes are so much higher. It's to account for non-responders.

  • I think this slightly misses the point that an ethnographic study is about observing and understanding people not statistically sampling their behaviour... – PhillipW Feb 18 '15 at 22:11
  • @PhillipW Fair enough. If you deal strictly with just observations though, I really don't think you're going to be able to get an objective answer for how many is enough. It really is to keep random sampling in groups until you're comfortable with things. – nightning Feb 18 '15 at 23:30
  • Agreed. Keep collecting data until you get to the point where you stop hearing new things. I'll put this as an answer. – PhillipW Feb 19 '15 at 9:48
  • BTW your answer - nightning - is spot on in terms of user testing and statistical significance. – PhillipW Feb 19 '15 at 9:53
  • 1
    @nightning: Thank you for this. I have zero experience with statistics and this was a good bit of information to spin me in that direction. I'm going to shoot for about six to start out with anyway based off of this response and my own experience based of usability testing. If I overshoot it and stop seeing new things after a few users then I can cancel some appointments, if I undershoot then hopefully I only have a few more to schedule to stop seeing new things. Sometimes it'd be nice to be able to pick to correct answers. Appreciate the help. – Fletchling Feb 19 '15 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.