Should I invite existing users to my summative usability test?

Question

I'm about to do a summative usability test on an app with a bunch of participants. I am convinced that I should mostly invite people unfamiliar with the app to see how well they are doing when encountering the interface for the first time.

However, I am also curious about how existing customers are using the app.

Would inviting existing users hurt the results of my summative usability test too much? Should I rather invite existing users for a separate test? Is there a benefit of including existing users?

Answer 1

Would inviting existing users hurt the results of my summative usability test too much?

Yes

Should I rather invite existing users for a separate test?

Yes

Is there a benefit of including existing users?

No

---

Think about this: SUT is designed to be ran on laboratory conditions, with a very specific set of guidelines oriented to define usability benchmarking, which is defined after rigurous statistical analysis. If you mix your audience between experienced and non-experience users, you'll certainly have a contaminated sample, because while not impossible, it will be very difficult to distinguish these users. Furthermore, running ANOVA will give you really unpredictable results. If you REALLY need to do it, better go with chi-square (and make sure you have a dimension to recognize experienced and novice users) or factorial ANOVA (take a look to an example of the use of factorial ANOVA on an HCI test pretty similar to yours, specially when it comes to the two ways you'll need to define), but it will always be more accurate and easy if you run 2 separate tests

Answer 2

Recruit the participants based on your user profiles or personas.

As per the process, selection of participants is based on the "user profiles" or "personas" that you have developed during research and data gathering.

If you change the participants during "Summative Testing", definitely that will impact the conversion rate and you will not get the accurate information about the system / application.