I am assigned to do usability test from a new bookstore website in mobile and desktop. Because of the limited resources, I need to be careful in recruiting participants to get the best result.

So I need to be sure: - Do I need to use different set of user (ex. 5 user for website in desktop and 5 user for mobile) or i can use the same set of user (ex. 5 user test both platform) - What is the pros and cons of using the same set of user to test a different platform of a product?

  • Are we assuming that you are only going to have between 5-10 user? How much time and resources are you likely to have? – Michael Lai Sep 12 at 22:24

With only 5 to 10 users overall, you will focus on qualitative analysis. The sample will be too small to do any kind of meaningful quantitative analysis.

To gather qualitative feedback, it is often useful to show 2 different options, and ask the testers to compare. You will get more in-depth and valuable feedback. This approach is even more valid here, as it's not 2 alternative designs for the same device, but for 2 different devices. You will be able to assess the consistency of the UI across platforms, and the transfer of familiarity from one to another. I would suggest to vary the order, start with mobile then desktop for the 1st group, and vice versa for the 2nd group (assuming that your users can start their journey with either device).

If you wanted to do a quantitative analysis with a larger sample, it would be preferable to have 2 separate groups, one for each device / option (and assign testers randomly to either group). If users test both devices, you will need to differentiate in the results tests on mobile seen first versus tests on mobile second, to see if the order had any impact.

I think you can consider the user to be different in a different test, even if he or she is physically the same person. Their role would be different (desktop user and mobile user). Especially if your resources are limited.

The Cons of using the same people for both platforms is that for the second time, they already have an insight of your application, they are not newcomers anymore. They may search similar thing, learned from the previous test.

Although, for me, this could be seen as an advantage, so you can see how a user might be led by one platform after switching to the other.

  • +1 Thanks for your contribution to UXSE! Knowing that using the same people for both platforms can be an issue, how might you control for this effect in your user testing? – Michael Lai Sep 12 at 22:24

User testing questions that involve limited resources are always tricky! But just like doing user research, you need to have an end goal or outcome in mind so you can best plan and construct your testing plan to make the most of those limited resources.

Here are some considerations that might help you answer your own question:

  1. If you are focusing on the difference between the desktop and mobile view of the website, there will be less variables to consider if you are using the same participant for both tests. But as someone pointed out it is possible to get a biased result if say you tested the desktop view on all users first and then mobile view second. So to control for that bias you can randomly assign half of the people to do the mobile view first and the rest to do the desktop view first.

  2. If you are focusing on the general usability across the desktop and mobile view of the website, I don't think you necessarily need to all the participants to go through the desktop and mobile view. It may simply take too much time (if you have lots of features for them to test) or too difficult to schedule in long sessions for all users, or not all users may complete the desktop and mobile view in the same way. If there are problematic issues then it will only take a small number of participants to reveal them (in general).

  3. If you are focusing on how seamless the experience is across desktop and mobile views of the website then it is pretty obvious that you need to get the same person to do both of the tests.

  4. If you want to compare the usability between the desktop and mobile views of the website and you have large enough numbers then you don't really need to rely on the same person doing both tests. If you are only looking for critical usability issues (and not subtle differences in usability or user experience), it is possible to do the comparison with relatively small number of users as well.

Hope this gives you plenty to think about in the complexity that is user testing :)

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.