User Experience Stack Exchange is a question and answer site for user experience researchers and experts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

A bit of backstory first: I'm setting up a usability test involving electronic medical records (EMRs). The test has to follow some draft testing guidelines (long pdf) set up by the National Institute of Standards and Technology (NIST) to identify usability issues that might impact patient safety. These draft standards are still somewhat flexible and open for comment.

My question comes from the requirements for selecting participants (pg 39):

Users are supposed to be familiar with EMRs but not the specific product they are testing.

I have two questions about this:

  • Isn't there an advantage to testing a mix of new and experienced users? New users can identify things that don't make sense but experienced users can reveal poor usability that could cause issues (e.g. error/warning blindness).

  • New users are supposed to receive training on the specific product immediately before they are tested. Is there a way to train users on a relatively complex product like this and not bias their usability testing immediately after?

share|improve this question
You could train them on a similar EMR (from different vendor). Thus they understand the concept of EMR but not your specific interface. – FrankL May 18 '12 at 17:27
up vote 5 down vote accepted

In Don't Make Me Think: A Common Sense Approach to Web Usability by Steve Krug, he notes some very interesting points that I believe are relevant to your question.

The importance of recruiting representative users is overrated. It's good to do your testing with people who are like the people who will use your site, but it's much more important to test early and often.

He goes on to mention something I strongly agree with in the section on recruit loosely and grade on a curve: "we're all beginners under the skin." If you look closely at an expert, you'll often find someone who's also muddling through -- just at a higher level.

As well, it's mentioned that it's usually not a good idea to deisgn a site so that only a very narrow view of your target audience can use it.

If you design a site for accountants using terminology that you think all accountants will understand, what you'll probably discover is that a small but not insignificant number of accountants won't know what you're talking about.

If a beginner can use it, an expert can. As well, experts are not often put off by something that is made clear enough for the someone who is new.

There are exceptions to these "rules of thumb" though, and if your site is intended to be used exclusively by one type of user and its no harder to recruit from that group, then there is no great reason not to do so. However, at the end of the day any testing is obviously better than no testing.

share|improve this answer
Another great resource on the topic is Krug's book Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. – Jessica May 18 '12 at 17:19
But the test is in the realms of medicine, so you have a narrow user base with knowledge of the topic. And tests have to follow strict rules as mentioned in the question. I think you aren't allowed to just grab someone who is passing by. For me it sounds the test are mandantory. – FrankL May 18 '12 at 17:24

First of all, I do agree with GotDibbs; it is MORE important to test early, and frequently throughout the process.

However, by not testing on "experts"/your real target group, you might miss out on some useful comments, questions etc. that only a professional would come up with. These comments could give you valuable insight when designing for your users' needs.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.