5

I'm going to be testing a way for students at the University of Leicester to print wirelessly to our printers from their own mobile device (could be smartphone, laptop, tablet etc.)

There are two different ways for them to print - they can send their document as an attachment in an email, or go to a website upload the file. I'll be testing both these methods against benchmarks, whilst also comparing them against eachother.

My problem is that our students will also likely want to use different kinds of devices to print (although the feedback we have had is that laptops and macbooks. are likely to be the most important). With time restrictions we can only realistically test around 10 students, and I'm struggling to see how we could test different devices with such a small sample (the type of device will inevitably influence things like time on task, so combining the scores on a task where participants used different devices is clearly a no-go)

The system isn't going to be changed much before release, and so the testing is going to be summative in nature - we want to see if it meets user requirements and establish a benchmark when future improvements are made. The iterative testing approach of going with a few users, then fixing issues and retesting isn't really relevant here.

Should I be concentrating only on one device for this study, or is there a way for me to be testing more than one device even with such a small sample, and still get valid results?

  • 10 participants is actually more than enough for such tests. Neilsen recommends no more than 5 really, so you could split your participants into 2 or 3 groups and have each group use a different device. – JonW Jul 31 '14 at 11:54
2

The often misquoted '5 users' is because Neilsen is describing 'iterative' testing of the same website.

*You want to run multiple tests because the real goal of usability engineering is to improve the design and not just to document its weaknesses. After the first study with 5 users has found 85% of the usability problems, you will want to fix these problems in a redesign.

After creating the new design, you need to test again. Even though I said that the redesign should "fix" the problems found in the first study, the truth is that you think that the new design overcomes the problems. But since nobody can design the perfect user interface, there is no guarantee that the new design does in fact fix the problems. A second test will discover whether the fixes worked or whether they didn't. Also, in introducing a new design, there is always the risk of introducing a new usability problem, even if the old one did get fixed.

Also, the second test with 5 users will discover most of the remaining 15% of the original usability problems that were not found in the first test. (There will still be 2% of the original problems left — they will have to wait until the third test to be identified.)*

However as Neilsen also states:

zero users give zero insights

So its clearly better to do some user testing with a small sample size and just accept that the results aren't going to be that accurate.

Even with 3 users per device type you'll learn a lot about user behaviour (the why's and how's if not anthing you can hang numbers on).

  • Thanks so much for your response! I should have said in the original message (now edited), but the testing is summative rather than formative (we arent going to be making any drastic improvements before its released) - we want to measure the results against usability objectives and set a benchmark when future improvements can be made (which unfortunately isn't likely to be soon), rather than take an iterative approach. Ideally I want to be able to hang numbers on the results, which is why I'm having a tricky time with this multiple devices thing – Joe Jul 31 '14 at 13:01
  • I don't think you can usefully hang numbers on the results with any degree of accuracy with this size of sample - you need to test more people if you want numbers. – PhillipW Aug 1 '14 at 13:38
1

Since your goal in the research is summative, you do not necessarily need to test multiple devices. You can choose what you are measuring, and be clear that this is what you are measuring.

If you do choose to measure a single type of device, you can either do it based on data (if you have it) about the types of devices that students currently use to print, or you can simply choose one. If you don't have this data available at your school, you might be able to ask system administrators of other similar universities if they have data that they could share with you. If you simply choose one, you can either do it based on assumptions (most people are printing Word documents, and that's most likely being done from a laptop), or you can do it based on where you think you have the most problems (which is usually an attempt to get those issues addressed in the future).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.