Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

What are your take on usability testing and responsive web sites? The flow of page objects, hiding and showing different types of menus or the resizing of objects could have a big impact on findability.

Let's say that you have worked with 3 break points and developed a small mobile version, a medium tablet version and a large desktop version. If you want to test the site on all these three versions, instead of the standard 6 test participants you now have to test it with 18 participants. The amount of work basically triples.

  • How should you test and with how many?
share|improve this question
    
I'm just curious: do you have a formal reference (book/paper/study) that states that six is the standard number of participants? –  CJ Franken Nov 22 '12 at 11:34
1  
Yes, Jakob Nielsen has written about it here. Even if he suggest using 5 participants: useit.com/alertbox/20000319.html –  Tony Bolero Nov 22 '12 at 11:41
    
Thanks. I see that he has a follow-up (June 2012) post about the same topic here: useit.com/alertbox/number-of-test-users.html . I was dubious at first (coming from a statistical analysis background), but I see that he has expanded his recommendation: Quantitative studies (at least 20 users), Card sorting (at least 15 users) and Eye tracking (at least 39 users) –  CJ Franken Nov 22 '12 at 11:58
add comment

3 Answers 3

up vote 3 down vote accepted

It depends what your goals are, and how you assess the different risks in what you're testing. Some things to consider:

  • How do your usage patterns break down? Are the majority of your users coming in on mobile devices? Then you should probably focus more of your testing resources on that breakpoint.

  • You could keep the same number of participants - but run the test multiple times on different devices. This saves you finding more people, and will let you find some problems. It will also likely introduce artefacts when users experience of one format will effect their expectations in another. This could be significant for some testing cases (e.g. first use).

  • Do the people using the site typically stick to one form factor? Do the mobile folk stay-mobile? Do the desktop folk stay on the desktop? Maybe you should be testing the same user on multiple devices, or maybe you need to recruit from two different segments and test each...

  • How many tests are you doing? How are the tests integrated into the rest of the product development process? If you're just doing one or two rounds of user testing then you probably need to be spend some time testing every configuration separately. If you're doing a test once a week feeding into an iterative product development process you can probably just pick a device at random and trust the repetition over time to even the feedback out.

  • What's the size of the pipe downstream from your usability tests. If there are only the resources to fix four issues, then their is likely little point you producing a list of forty. Adjust the scope of testing accordingly.

share|improve this answer
add comment

You still can do it with the same number of participants as you would do with a single test -- it just takes longer to test each user because you're showing them more material (i.e. three breakpoints instead of just one). However, you should "shuffle" and randomize the order in which you test each breakpoint so you don't skew the results due to familiarity (which should increase with each successive test the user does).

This is likely more similar to how a user will encounter your site in the real world anyway -- finding the site in one version, and migrating to another.

share|improve this answer
    
+1 for shuffling the test order –  Justin Jan 2 '13 at 23:43
add comment

My intuition says that you should have separate tests for the different break points. If we only consider two scenarios (small-mobile, large-desktop) then you are targeting two distinct input devices (touch vs mouse). You may find that your tests look different for touch devices, than for desktop devices (the tasks may differ, or priorities change depending on device).

As to how many participants, I defer to the research done by Nielsen (as provided in your comments to the original question).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.