A bit of backstory first: I'm setting up a usability test involving electronic medical records (EMRs). The test has to follow some draft testing guidelines (long pdf) set up by the National Institute of Standards and Technology (NIST) to identify usability issues that might impact patient safety. These draft standards are still somewhat flexible and open for comment.
My question comes from the requirements for selecting participants (pg 39):
Users are supposed to be familiar with EMRs but not the specific product they are testing.
I have two questions about this:
Isn't there an advantage to testing a mix of new and experienced users? New users can identify things that don't make sense but experienced users can reveal poor usability that could cause issues (e.g. error/warning blindness).
New users are supposed to receive training on the specific product immediately before they are tested. Is there a way to train users on a relatively complex product like this and not bias their usability testing immediately after?