Questionnaire for Small Sample Studies (5-7 users)

Question

I am currently working on my thesis and am working on conducting an user research about chatbot experience with real users. Currently, I will test three different chatbot prototypes to test:

(1) For each prototype, I will find around 5-7 users to test it

(2) To test each prototype, I will first interview them (to know their usage of chatbot, background, etc.), then usability testing (giving them tasks), and finally I wan to use some metrics to measure how good is the prototype.

(3) I wonder if anyone recommends any metrics for measuring the usability of a chatbot app? I currently planned to use the SUS questionnaire, which I found a few theses using it to evaluate chatbot, but I am not sure if it works with a small number of samples (5 users).

Does anyone recommend some different approaches?

Answer 1

TLDR; There's a a prior question which I suggested a couple of methodologies but the resources below has way larger alternatives.

I conducted a thesis research as well and started like writing down all the methodologies and metrics first from Tullis & Albert, 2013. There are a couple of more sources below I can share if you'd like,

• Hartson & Pyla, 2012
• Allanwood & Beare, 2014
• Goodman et al., 2012

The method of the research and depending metrics should be shaped according to this specific method. So the thing is you should research all of them to fully understand what you need and what you don't at least to comprehend all the techniques you may want to make use of.

It's more easier to get the rest of the job done, once you're comfortable with all of the possibilities even it's not an easy start. Otherwise it'd be just a suggestion from some other people you might not fully understand the aim of choosing that specific methodology.

Answer 2

When conducting usability tests, task completion rates combined with a standard usability measurement is typically enough data for me. For the standard usability measurement, the SUS definitely works, but I often turn to the UMUX-lite instead. It is only 2 questions and can be converted to an SUS score reliably.

As far as how many users you need for the SUS, there's a good answer in this thread: How many users do I need to run a SUS and get valid result?