Explaining small sample size---influencing stakeholders

Question

I am new to the user experience world. Based on some references I've read and threads here, it's Ok to determine statistical significance and compare means for a small sample size of 11 (7 users in group A and 4 users in group B). How do I influence stakeholders, peers that my sample size is acceptable to do stats in the first place? Is there a more specific statement or explanation that will catch their attention. It seems that although there are experts opinion that says "it's enough sample size", most experienced usability professionals do not agree.

Answer 1

I know people quote Nielsen a lot here and some love him and some hate, but in any case, Nielsen has a lot to say about the benefits you can get from testing with a very small sample size. He also has some interesting points on why a small amount of data is better than guessing, which is what usually happens if no testing is done at all.

The theory, I think, goes along the lines of: humans are much more alike in their behaviours to stimuli than you would realise and so testing with a small sample size still gives you a lot of useful results. Further more, testing with larger samples hits a level point in usefulness very quickly because you are simply getting more subjects that are likely to behave very much the same or in a narrow range.

Nielsen quotes the numbers 5 as being the point at which you start to get proportionally less new information for additional subjects, and 15 as the point where it pretty much levels off.

I don't know if these numbers are right, but your question was not asking for what those numbers were, but rather for something to show the stakeholders to convince them that your small sample size is still worth it. I should think that showing them the pages I have referenced here and then getting them to Google Jacob Nielsen (>1.25 M results) should do it ;)

Answer 2

This topic comes up quite regularly:

A quick search on here finds:

When does statistical significance matter?

Ideal number of candidates for user-testing

Personally I wouldn't even mention 'statistical significance'.

Usability testing is a form of qualitative market research, rather than being quantitative scientific research. We're not doing a scientific experiment: we're looking for meanings.

You don't just want to know how many 'observed behaviours' there were - you want to know 'why?'. And often lots of little 'whys' turn out to just part of one big user misconception.

Answer 3

Steve Krug's book 'Rocket Surgery Made Easy' is a very relevant book here. (Steve Krug's site - http://www.sensible.com/rocketsurgery/index.html - easy to find on amazon which has inside preview)

Quoting a few sections of the book which I thoroughly recommend reading.

---

What do I tell people who say, “But if you’re only testing three people at a time, it can’t be statistically valid. You can’t prove anything that way”? Here’s what you should say to them: “You’re absolutely right. Testing with so few people can’t possibly produce statistically valid results. The samples are way too small to even bother with statistics. But the point of this kind of testing isn’t to prove anything; the point is to identify major problems and make the thing better by fixing them. It just works, because most of the kinds of problems that need to be fixed are so obvious that there’s no need for ‘proof.’ ”

Try to say it with a lot of conviction and a friendly smile.

---

Three is enough The debate over how many test participants you need has raged for a long time in the usability community, like one of those coal mine fires that burn underground for decades. Almost everyone agrees that there are diminishing returns from having more users do the same tasks: the more users you watch, the fewer new problems you see. Most of the research that’s been done—and the arguing—is about how many users will uncover most of the usability problems in what you’re testing. For instance, “Testing with five users will find 85 percent of the problems.” But that’s the wrong argument for you, the do-it-yourselfer. You’re not interested in what it takes to uncover most of the problems; you only care about what it takes to uncover as many problems as you can fix. After many years, I’ve settled on three users in each round of testing for a number of reasons:

• The first three users are very likely to encounter many of the most significant problems related to the tasks you’re testing.
• Finding three participants is less work than finding more.
• It’s much more important to do more rounds of testing than to wring everything you can out of each round.
• Testing with just a few users makes it easier to do more rounds.
• Testing with three users makes it possible to test and debrief in the same day.
• With only a few users, it’s easier to encourage people to come and observe.
• In addition to diminishing returns, there’s the tedium factor—for you as facilitator and for the observers. Starting with the fourth user of the day, there’s usually a lot more snack eating, checking of voice mail, and side conversations.
• When you test with more than three at a time, you often end up with more notes than anyone has time to process—many of them about things that are really “nits.” This can make it harder to see the most serious problems—the “can’t see the forest for the trees” effect.
• Testing with a lot of users can uncover an overwhelming and dispiriting amount of problems. Prioritizing and triaging them becomes a problem in itself, another process to manage.

Answer 4

I’m puzzled myself by this notion some people have that sample sizes can be too small for inferential statistics. Unless you literally get divide-by-zero, there is simply no such thing. I would explain to anyone who questions using statistics that the whole purpose of calculating the p-value is to determine if the sample size, however small it is, is large enough to draw a reasonable conclusion. Refusing to do statistics but using the results anyway does not solve the problem of having a small sample size. Saying a sample is too small to bother with statistics is like saying a boat is too small to worry about overloading it.

As for getting your team and stakeholders to accept your statistical results from a small sample size, UX peers shouldn’t be a problem because they should have at least a rudimentary statistics background and understand that the p-value is what’s important, not the sample size. If they don’t understand that, then feel free to direct them to my Stat 101 article.

For stakeholders, assuming they have any quantitative sense at all, try to report the p-value in terms they understand. For example, for a p of 0.018 when comparing completion times, try something like “I’ve done the probability calculations, and even with this small sample size, we can be 98.2% sure that Design A is faster to use on average than Design B for users in general.” I would use 3 decimal places. Saying “I’m 95% sure” or “99% sure” sounds like you’re making up numbers. Emphasize that your calculations take into account the sample size. It’s literally in the equation.

I would report the p-value (or one-minus-the-p-value), not whether it’s “statistically significant” or not, because “significance” takes some mental gymnastics involving null hypotheses that makes it harder to grasp. Stakeholders may confuse statistical significance with business significance. Reporting the p-values instead makes it clearer what stakeholders are choosing to do. It may encourage stakeholders to embrace your findings even when they are not technically significant, but I would argue that may be legitimate to do outside of academic research. For example “89.1% sure A out performs B” sounds like a pretty good bet for a business. It really depends on how comfortable your stakeholders are with an 11% chance of being wrong (of making Type I error). Giving the p-value let’s them make an informed decision.

For stakeholders that aren’t so quantitative, but still don’t trust small sample sizes, try to give them an intuition of the role of the magnitude of the effect. With small sample sizes, you’re going to observe a dramatic difference in A and B if you’re going to have anything like a small p-value. Say something like, “Look, Design A was completed in about half the time of Design B on average. Now with this small sample size, I can’t say exactly how much better Design A is (at this point you could give confidence intervals), but it’s really unlikely that A and B are actually same on average when you’re seeing this kind of difference.”

If that doesn’t work, show a video of one average user from each group. Seeing one user thrash around on B and another breeze through A will hit home for some stakeholders far better than all the numbers in the world.

BTW if you can get closer to equal numbers in each of your groups (i.e., 6 and 5, rather than 7 and 4) you’ll get more statistical power. Power depends more on the size of the smaller group than the larger one.

Answer 5

It's a qualitative vs. quantitative issue. Both are important, both can produce slightly different data. You really want both whenever you can get it. In the past we've done on-site observed user testing of 7-10 people and then may do an online test of 30-100 people.

If stakeholders are concerned about low sample sizes, then they need to up the budget and staff so you can do more testing.