0

I ran 2 user testing with a 3 months gap between. In the first one, I've tested the current design and functionality of the app with 6 scenarios. The later one, I conducted the test with new design prototype by utilising same scenarios.

I used same participants in both study. Is it harmless to use same participants or would it be better to use same profile but different participants?

Improve this question
1

2 Answers 2

Reset to default
3

It seems you are doing iterative design. It's fine to use the same participants because the purpose of the study is to identify areas for improvement. However...

  • The more often you use the same participants, the more specific your design will become to their preferences. It may be very much to their liking, but not good for anyone else.

  • Users may become exhausted from testing. They may start nitpicking unimportant details, much as developers sometimes do when working on the same project for extended periods.

  • Users may begin picking up on tester preferences, telling you what they think you want.

If your user pool is inherently limited, you can consider drawing random subsets for testing.

If doing bonafide experiments (to compare interfaces), crossover design is valid methodology. You still need enough participants to ensure statistical power. You would need two complete designs. Preferably they would differ in just one aspect. Randomize users to one design or another. Switch designs when tasks are complete. Use comparable, but different, tasks to avoid training effects.

Improve this answer
7
  • Thanks @xiota! Very useful indeed. I had 2 purposes once was as you said looking for areas to improve, second was benchmark design with the previous version. Is it still acceptable to use same participants? What is your opinion on comparison studies?
    – Izzie
    May 14, 2019 at 9:43
  • I think usability studies for iterative design and comparison studies are very different. Usability studies for iterative design are screening tools. Comparison studies are like confirmation tests. Think of it like getting a pap smear. Does a positive screen mean you definitely have cancer? No. You need confirmation. Can you use a screening test for confirmation? No. A "better" test is needed. Pick the appropriate study for your purpose.
    – 習約塔
    May 14, 2019 at 9:54
  • I think it's desirable to have a larger pool of participants, regardless of study, just so you don't get false positives from over-training and other biases (making the design too specific to a non-representative sample of users).
    – 習約塔
    May 14, 2019 at 9:57
  • I got your point however, I'm still a bit skeptical about the reason of why we cannot do both in the same test. My main purpose was to compare the design -that we have built from the feedback of previous test- with the previous version. In the meantime some issues have been detected so it gives me also improvement points. I used post task questionnaires and SUS for the comparison. In order to utilise these I used scenarios and during the scenario execution I captured some problems.
    – Izzie
    May 14, 2019 at 10:06
  • I agree though I should have gathered a bigger sample population.
    – Izzie
    May 14, 2019 at 10:06
0

To be totally precise, you should use different participants. They should not have any emotional attachment or know you, or your brand. As it can affect their answers and therefore objectivity of the results.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.