Running usability tests with the same people that have passed it previously

Question

I will be doing the second testing soon. The question is: Is it possible to invite to the second testing those people who passed the first testing?

Answer 1

No.

People learn.

If the software is similar to the first time, your users will have learnt how it works.

Answer 2

Yes, but their input should be treated differently than users who were not a part of the first round, and you certainly still require a group of users who have not previously tested it.

To expand as requested in the comments, results from users of a previous test should be strictly separate from new users, and will be handled more anecdotally. i.e. a user in the second test may report back that "the options selection feature works, and I was able to use it the first time, but on the second use, noticed it's very tedious when you wish to turn off or on all options. The process could be simplified by adding a "toggle all" switch at the top of the group". The feedback could be anything- the point being a second time tester may find additional details a first time tester never will. It should change or affect your goal %'s for a first time usability test, but more feedback rarely hurts.

Answer 3

In learning we have received from PwC Experience Center on Qualitative Usability testing, it was noted participants should be new ones.

Advantages of this are:

• if they are first time here, they are acting as real users, and not design test professionals
• they don't have any experience with the brand or app (positive or negative), therefore they are more honest and precise with giving answers after
• it's statistically not totally correct to mix findings from reused and new participants

Therefore, I would say it's possible (as it is practical sometimes). But data taken from tests should be taken with a grain of salt.

Answer 4

It depends on which market you're trying to test.

We can distinguish the following market segments, and probably many more:

1. users who have never experienced a product in this category
2. users of a competitive product who have never experienced your product
3. users of a competitive product who have experienced your product
4. active users of your product
5. disengaged users of your product for 1+ weeks
6. disengaged users of your product for 1+ months, etc.

All of these groups might be important to test, depending on what your goals are. For example, if you're trying to figure out how to re-engage inactive users, you will probably focus your testing to optimize group 5, but also test other groups to validate that you haven't deteriorated their experience.

Back to your question: can I invite the same participants back for a second round of testing?

Of course you can. The problem is that your first test might have moved that user into a different market segment. For example, if your goal is to test users from market segment 1 — users who have never experienced a product in this category, then the participants from the first round of testing have now experienced a product in this category and are somewhat familiar with it, so they now belong to segment 2 or 3, depending on how much you've revised your UX.

Answer 5

In my opinion this is fine as long as the 1st test doesn't have any learnings that could be used in the 2nd test (unless you want that). There can be also some benefits as they are more familiar and comfortable with you they are more likely to share more. If you are conducting user testing regularly it's likely to happen as some participants like the fact their feedback is being used to make improvements.