TLDR: Sample size required is highly dependent on what you're trying to measure and study conditions. In an ideal environment, you need 6 per group for small number of groups for statistical significance. In a more typical study, you probably want 10-12 per group. If you don't care as much about statistical significance, by all means use less.
How do you determine sample size?
There's 3 main factors that affects sample size for obtaining statistically relevant results when you're making comparisons between groups.
- Type of data you're collecting and the size of the difference between the groups you're trying to observe. This affects: statistical power, your ability to detect a difference should it exists between groups. The more precise the measuring method and the larger the overall difference between the group in relation to the preciseness of your measuring method, the smaller the sample size you'll need per group. Some tests are also inherently better at detecting differences. E.g. if you're testing to see if X is larger than Y, but don't care if X is smaller than Y (1-tailed test), then your statistical power is twice as large as if you want to see if X is different, larger or smaller, than Y (2-tailed test). Data may also be continuous (e.g. intensity of color) or discrete (e.g. Is either Red or White). Tests for continuous distribution are inherently more powerful than discrete tests and requires smaller sample sizes.
- How confident you must be that the difference you observe truly exists and it didn't show up by chance. Sometimes people refer to this in the reverse as "margin of error". You'll need larger sample size per group if you need more confidence.
- How much variability is there in what you're trying to measure. The more "noise" you have compared to "signal strength", the larger the sample size you need. Behavioural studies tends to have way of variability as compared to physical studies with objects.
But I'm sure what you're hoping for are actual numbers...
So here's an example of a properly controlled, and randomized study in a lab.
If you are talking about observing a difference in 1 variable statistically (T-test) for a medium (~20-30%) difference in a controlled behavioural experiment, you need 6 in each sample to observe this difference 80% of the time. The numbers required for an analysis of variance (ANOVA) for 3 or 4 groups are around the same as well. If you have more groups, or the difference you're trying to observe is smaller, you'll need more per group. I don't have the statistics textbooks in front of me to quote stuff. But this is commonly used for pilot behavioural studies in drug research.
When you deal with field studies, you'll have way more variability. I don't know how much more in your case, but a typical power vs sample size tends to be an exponential curve. There's a point in which you get very little extra value per additional tester you add.
So a ballpark "reasonable" number (I'm pulling this out of my ass for a typical semi-controlled study where you need for statistical significance results) is 10-12 per group. Above 20, you're usually not getting much bang for your buck for most readily observable difference between groups.
Also, groups of unequal sizes decrease sensitive of tests. i.e. a test with groups of 6 & 6 have much higher statistical power as groups with 5 & 7.
There's an excellent article on Measuring U on how to determine your sample size. Note: their example assumes they're looking at survey results with a completion rate of only 50-70%. That's why their sample sizes are so much higher. It's to account for non-responders.
