User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
I am assigned to do usability test from a new bookstore website in mobile and desktop. Because of the limited resources, I need to be careful in recruiting participants to get the best result.
So I need to be sure: - Do I need to use different set of user (ex. 5 user for website in desktop and 5 user for mobile) or i can use the same set of user (ex. 5 user test both platform) - What is the pros and cons of using the same set of user to test a different platform of a product?
With only 5 to 10 users overall, you will focus on qualitative analysis. The sample will be too small to do any kind of meaningful quantitative analysis.
To gather qualitative feedback, it is often useful to show 2 different options, and ask the testers to compare. You will get more in-depth and valuable feedback. This approach is even more valid here, as it's not 2 alternative designs for the same device, but for 2 different devices. You will be able to assess the consistency of the UI across platforms, and the transfer of familiarity from one to another. I would suggest to vary the order, start with mobile then desktop for the 1st group, and vice versa for the 2nd group (assuming that your users can start their journey with either device).
If you wanted to do a quantitative analysis with a larger sample, it would be preferable to have 2 separate groups, one for each device / option (and assign testers randomly to either group). If users test both devices, you will need to differentiate in the results tests on mobile seen first versus tests on mobile second, to see if the order had any impact.
I think you can consider the user to be different in a different test, even if he or she is physically the same person. Their role would be different (desktop user and mobile user). Especially if your resources are limited.
The Cons of using the same people for both platforms is that for the second time, they already have an insight of your application, they are not newcomers anymore. They may search similar thing, learned from the previous test.
Although, for me, this could be seen as an advantage, so you can see how a user might be led by one platform after switching to the other.
User testing questions that involve limited resources are always tricky! But just like doing user research, you need to have an end goal or outcome in mind so you can best plan and construct your testing plan to make the most of those limited resources.
Here are some considerations that might help you answer your own question:
If you are focusing on the difference between the desktop and mobile view of the website, there will be less variables to consider if you are using the same participant for both tests. But as someone pointed out it is possible to get a biased result if say you tested the desktop view on all users first and then mobile view second. So to control for that bias you can randomly assign half of the people to do the mobile view first and the rest to do the desktop view first.
If you are focusing on the general usability across the desktop and mobile view of the website, I don't think you necessarily need to all the participants to go through the desktop and mobile view. It may simply take too much time (if you have lots of features for them to test) or too difficult to schedule in long sessions for all users, or not all users may complete the desktop and mobile view in the same way. If there are problematic issues then it will only take a small number of participants to reveal them (in general).
If you are focusing on how seamless the experience is across desktop and mobile views of the website then it is pretty obvious that you need to get the same person to do both of the tests.
If you want to compare the usability between the desktop and mobile views of the website and you have large enough numbers then you don't really need to rely on the same person doing both tests. If you are only looking for critical usability issues (and not subtle differences in usability or user experience), it is possible to do the comparison with relatively small number of users as well.
Hope this gives you plenty to think about in the complexity that is user testing :)
