0

I'm a big fan of Design Sprints, especially of their value to efficiently align a team on the problem and proposed hypothesis/solution. There's one aspect with which I've always struggled: testing.

On the last day, the team tests their idea with 5 participants. This sample size is cited as sufficient per Nielsen Norman Group's study on the sample size needed to achieve optimal usability testing results (i.e., identify most usability problems).

But is the research objective on Day 5 of a Design Sprint truly a usability test? I don't think so. The objective, to me, is more of a concept test: does this proposed solution solve your meaningful problem? (And not: is this proposed solution intuitive to use to achieve your goal for a specific real world scenario?)

Example: On the last day of the Design Sprint, say a team tests the hypothesis that a flying car can solve the need of frustration of sitting in traffic. It's testing if the flying car meets the need as expected by target customers (as opposed to a hyperloop or super sonic hedgehog). It's not testing whether using the flying car - i.e., opening the door, starting it up, managing it through the flight, etc. - has usability problems that need resolved.

Is it usually acceptable to test a concept with 5 participants? What is an ideal sample size to test a concept?

  • Not sure why someone voted to close this question. It's clear, informed and has possible answers. Anyways, @RPJUX, try editing making a clear question to avoid it to be closed – Devin Jul 13 '18 at 18:56
1

But is the research objective on Day 5 of a Design Sprint truly a usability test?

You're correct. It's not (necessarily). It could be depending on the kind of sprint you do, the goals you define and the previously defined steps on that sprint. But you could be also testing a concept (just as you say).

The example you give, while extreme, is perfect: you can test the concept of flying cars as a solution for such problem. You don't need to (as a matter of fact you MUST NOT) test the flying cars themselves. (Off topic-ish note: if you do an usability test for a flying car with 5 people, I would call you a criminal).

A real life example is an study we did about accessibility for disabled people in Buenos Aires train stations. The findings were compared to previous interviews with people with disabilities and their families, and then a second round of interviews in which we presented possible solutions. These solutions were of very different types. For example, elevators. In this case, we tested the concept of having elevators with specific features, just not the elevators themselves.

In short: your final testing could be any of usability, concept, analytical (ie: results from A/B or multivariate testing, or ANOVA, or whatever), preference, emotional testing, user preference or whatever serves to the sprint goal. Or in other words: it's just a method to extract results using any given mothodology.

PS:if interested, you can see a small chronicle of what we found, in Spanish

  • Thank you! I agree with your perspective very much. For your study, what sample size did you use to test concepts? I've yet to find a trusted resource on sufficient sample size to test if a concept satisfactorily meets identified need (as opposed to a usability test to see if a design is intuitive to use to meet a goal). – RPJUX Jul 13 '18 at 19:47
  • I didn't do the interviews, but if remember correctly, we used 14 people. With a caveat: some were people with disabilities, some were family that had to carry people on wheelchairs. Don't remember the exact proportion but think 50/50. Also, we had a scale that put each of them on some specific level according to their degree of disability. All in all, as a general answer, for any kind of test that is NOT usability, we try to test with at least 20 people for early stages. We do a lot of focus and preference tests, so 5 is not enough for that purpose, we require different ages, social, etc. – Devin Jul 13 '18 at 20:29
0

The '5 participants' figure is often quoted out of context:

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

The '5 participants' are one iteration of a series of tests.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.