Usability testing and responsive web design

Question

What are your take on usability testing and responsive web sites? The flow of page objects, hiding and showing different types of menus or the resizing of objects could have a big impact on findability.

Let's say that you have worked with 3 break points and developed a small mobile version, a medium tablet version and a large desktop version. If you want to test the site on all these three versions, instead of the standard 6 test participants you now have to test it with 18 participants. The amount of work basically triples.

• How should you test and with how many?

Answer 1

It depends what your goals are, and how you assess the different risks in what you're testing. Some things to consider:

• How do your usage patterns break down? Are the majority of your users coming in on mobile devices? Then you should probably focus more of your testing resources on that breakpoint.

• You could keep the same number of participants - but run the test multiple times on different devices. This saves you finding more people, and will let you find some problems. It will also likely introduce artefacts when users experience of one format will effect their expectations in another. This could be significant for some testing cases (e.g. first use).

• Do the people using the site typically stick to one form factor? Do the mobile folk stay-mobile? Do the desktop folk stay on the desktop? Maybe you should be testing the same user on multiple devices, or maybe you need to recruit from two different segments and test each...

• How many tests are you doing? How are the tests integrated into the rest of the product development process? If you're just doing one or two rounds of user testing then you probably need to be spend some time testing every configuration separately. If you're doing a test once a week feeding into an iterative product development process you can probably just pick a device at random and trust the repetition over time to even the feedback out.

• What's the size of the pipe downstream from your usability tests. If there are only the resources to fix four issues, then their is likely little point you producing a list of forty. Adjust the scope of testing accordingly.

Answer 2

My intuition says that you should have separate tests for the different break points. If we only consider two scenarios (small-mobile, large-desktop) then you are targeting two distinct input devices (touch vs mouse). You may find that your tests look different for touch devices, than for desktop devices (the tasks may differ, or priorities change depending on device).

As to how many participants, I defer to the research done by Nielsen (as provided in your comments to the original question).

Answer 3

You still can do it with the same number of participants as you would do with a single test -- it just takes longer to test each user because you're showing them more material (i.e. three breakpoints instead of just one). However, you should "shuffle" and randomize the order in which you test each breakpoint so you don't skew the results due to familiarity (which should increase with each successive test the user does).

This is likely more similar to how a user will encounter your site in the real world anyway -- finding the site in one version, and migrating to another.