Testing with new vs. experienced users

Question

A bit of backstory first: I'm setting up a usability test involving electronic medical records (EMRs). The test has to follow some draft testing guidelines (long pdf) set up by the National Institute of Standards and Technology (NIST) to identify usability issues that might impact patient safety. These draft standards are still somewhat flexible and open for comment.

My question comes from the requirements for selecting participants (pg 39):

Users are supposed to be familiar with EMRs but not the specific product they are testing.

I have two questions about this:

• Isn't there an advantage to testing a mix of new and experienced users? New users can identify things that don't make sense but experienced users can reveal poor usability that could cause issues (e.g. error/warning blindness).

• New users are supposed to receive training on the specific product immediately before they are tested. Is there a way to train users on a relatively complex product like this and not bias their usability testing immediately after?

Answer 1

In Don't Make Me Think: A Common Sense Approach to Web Usability by Steve Krug, he notes some very interesting points that I believe are relevant to your question.

The importance of recruiting representative users is overrated. It's good to do your testing with people who are like the people who will use your site, but it's much more important to test early and often.

He goes on to mention something I strongly agree with in the section on recruit loosely and grade on a curve: "we're all beginners under the skin." If you look closely at an expert, you'll often find someone who's also muddling through -- just at a higher level.

As well, it's mentioned that it's usually not a good idea to deisgn a site so that only a very narrow view of your target audience can use it.

If you design a site for accountants using terminology that you think all accountants will understand, what you'll probably discover is that a small but not insignificant number of accountants won't know what you're talking about.

If a beginner can use it, an expert can. As well, experts are not often put off by something that is made clear enough for the someone who is new.

There are exceptions to these "rules of thumb" though, and if your site is intended to be used exclusively by one type of user and its no harder to recruit from that group, then there is no great reason not to do so. However, at the end of the day any testing is obviously better than no testing.

Answer 2

First of all, I do agree with GotDibbs; it is MORE important to test early, and frequently throughout the process.

However, by not testing on "experts"/your real target group, you might miss out on some useful comments, questions etc. that only a professional would come up with. These comments could give you valuable insight when designing for your users' needs.