Design Sprint - Sample SIze

Question

I'm a big fan of Design Sprints, especially of their value to efficiently align a team on the problem and proposed hypothesis/solution. There's one aspect with which I've always struggled: testing.

On the last day, the team tests their idea with 5 participants. This sample size is cited as sufficient per Nielsen Norman Group's study on the sample size needed to achieve optimal usability testing results (i.e., identify most usability problems).

But is the research objective on Day 5 of a Design Sprint truly a usability test? I don't think so. The objective, to me, is more of a concept test: does this proposed solution solve your meaningful problem? (And not: is this proposed solution intuitive to use to achieve your goal for a specific real world scenario?)

Example: On the last day of the Design Sprint, say a team tests the hypothesis that a flying car can solve the need of frustration of sitting in traffic. It's testing if the flying car meets the need as expected by target customers (as opposed to a hyperloop or super sonic hedgehog). It's not testing whether using the flying car - i.e., opening the door, starting it up, managing it through the flight, etc. - has usability problems that need resolved.

Is it usually acceptable to test a concept with 5 participants? What is an ideal sample size to test a concept?

Answer 1

But is the research objective on Day 5 of a Design Sprint truly a usability test?

You're correct. It's not (necessarily). It could be depending on the kind of sprint you do, the goals you define and the previously defined steps on that sprint. But you could be also testing a concept (just as you say).

The example you give, while extreme, is perfect: you can test the concept of flying cars as a solution for such problem. You don't need to (as a matter of fact you MUST NOT) test the flying cars themselves. (Off topic-ish note: if you do an usability test for a flying car with 5 people, I would call you a criminal).

A real life example is an study we did about accessibility for disabled people in Buenos Aires train stations. The findings were compared to previous interviews with people with disabilities and their families, and then a second round of interviews in which we presented possible solutions. These solutions were of very different types. For example, elevators. In this case, we tested the concept of having elevators with specific features, just not the elevators themselves.

In short: your final testing could be any of usability, concept, analytical (ie: results from A/B or multivariate testing, or ANOVA, or whatever), preference, emotional testing, user preference or whatever serves to the sprint goal. Or in other words: it's just a method to extract results using any given mothodology.

PS:if interested, you can see a small chronicle of what we found, in Spanish

Answer 2

The '5 participants' figure is often quoted out of context:

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

The '5 participants' are one iteration of a series of tests.