Can I test different devices when I'll only have a sample of around 10 participants?

Question

I'm going to be testing a way for students at the University of Leicester to print wirelessly to our printers from their own mobile device (could be smartphone, laptop, tablet etc.)

There are two different ways for them to print - they can send their document as an attachment in an email, or go to a website upload the file. I'll be testing both these methods against benchmarks, whilst also comparing them against eachother.

My problem is that our students will also likely want to use different kinds of devices to print (although the feedback we have had is that laptops and macbooks. are likely to be the most important). With time restrictions we can only realistically test around 10 students, and I'm struggling to see how we could test different devices with such a small sample (the type of device will inevitably influence things like time on task, so combining the scores on a task where participants used different devices is clearly a no-go)

The system isn't going to be changed much before release, and so the testing is going to be summative in nature - we want to see if it meets user requirements and establish a benchmark when future improvements are made. The iterative testing approach of going with a few users, then fixing issues and retesting isn't really relevant here.

Should I be concentrating only on one device for this study, or is there a way for me to be testing more than one device even with such a small sample, and still get valid results?

Answer 1

The often misquoted '5 users' is because Neilsen is describing 'iterative' testing of the same website.

*You want to run multiple tests because the real goal of usability engineering is to improve the design and not just to document its weaknesses. After the first study with 5 users has found 85% of the usability problems, you will want to fix these problems in a redesign.

After creating the new design, you need to test again. Even though I said that the redesign should "fix" the problems found in the first study, the truth is that you think that the new design overcomes the problems. But since nobody can design the perfect user interface, there is no guarantee that the new design does in fact fix the problems. A second test will discover whether the fixes worked or whether they didn't. Also, in introducing a new design, there is always the risk of introducing a new usability problem, even if the old one did get fixed.

Also, the second test with 5 users will discover most of the remaining 15% of the original usability problems that were not found in the first test. (There will still be 2% of the original problems left — they will have to wait until the third test to be identified.)*

However as Neilsen also states:

zero users give zero insights

So its clearly better to do some user testing with a small sample size and just accept that the results aren't going to be that accurate.

Even with 3 users per device type you'll learn a lot about user behaviour (the why's and how's if not anthing you can hang numbers on).

Answer 2

Since your goal in the research is summative, you do not necessarily need to test multiple devices. You can choose what you are measuring, and be clear that this is what you are measuring.

If you do choose to measure a single type of device, you can either do it based on data (if you have it) about the types of devices that students currently use to print, or you can simply choose one. If you don't have this data available at your school, you might be able to ask system administrators of other similar universities if they have data that they could share with you. If you simply choose one, you can either do it based on assumptions (most people are printing Word documents, and that's most likely being done from a laptop), or you can do it based on where you think you have the most problems (which is usually an attempt to get those issues addressed in the future).