Planning on running usability tests with a product that has an extensive on-boarding tutorial/process. Of course the actual product will allow the users to skip such steps however the product it self does require some technical knowledge to use.

The real users of the application will have gone through the on-boarding process and have some prior knowledge of what they need to achieve. On the other hand,the testers will lack some knowledge.

I see some options but I'm unsure what's the best one.

  1. Make the test simpler but include the on-boarding process for the specific area in the product.
  2. Cater for the lacking on-boarding process by covering the content in written instructions within the test.
  3. Go wild and test without on-boarding and see what happens anyway.
  • Set yourself how long the test should be. Let' say an hour. And then test the test on somebody else to see how what time the person will need and what instructions they'll need. It should help you to decide.
    – digsrafik
    Nov 19 '15 at 14:10
  • Does on-boarding mean training? The product requires special knowledge to use? Nov 19 '15 at 14:18
  • @KenMohnkern , yes this is a very specific product for specific users. The onboarding is used to give the users a tour of the application of where things are and when to use them. Nov 19 '15 at 14:35

Only three choices? I can think of another.

Keep it realistic

When I do a usability study that involves observing participants while using a product, I like to get the scenario as realistic as possible. And I go out of my way not to lead the participants.

For example, to I want to assess whether participants can complete a task that requires them to use a command on the Tools menu, the scenario—or instructions—that I ask each participant to read before they begin will not have the word "tool" in it.

For your product, your real users would typically walk through a tutorial. This could definitely be seen as "leading the participant." But I think the key questions in your case are how long is the interval between the tutorial and product usage, and how quickly do they complete the tutorial? Do they skip through it? Are they usually interrupted?

Then consider: could you reproduce or mimic the real situation in your research?

During each research session, perhaps you can start with a tutorial—be sure to rush your participants so they skim, or to deliberately interrupt them, as would occur for real users. If you ask them to do the tutorial immediately, before you discuss the process and before you record anything, then you'll have several activities that you can use to lengthen the duration between the tutorial and the product use:

  • "Please read and sign the copyright waiver and non-disclosure agreement."
  • "Please fill in this questionnaire" about the professional domain but unrelated to the features you're testing, or provide some other distraction task.
  • "Please read your email for 5 minutes while I set up."

And then carry on with the usability research of the product, the study's meat and potatoes that you want to record.

It'll take extra time, so be prepared to compensate accordingly, and schedule accordingly. In my opinion, it's better to test with fewer participants more realistically than to test with more users who have been cued by the tutorial.

To answer your question, I choose option 4, which you did not list. And that is: mimic real usage as much as possible.

  • 1
    Thanks, after reading your answer I do think it's the better way. I will just increase the time of my current tests and give the users a more real scenario. Nov 19 '15 at 14:37

In addition to the recommendations by @JeromeR...

According to Nielsen (http://www.nngroup.com/articles/usability-101-introduction-to-usability/), usability is defined by 5 quality components:

  • Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
  • Efficiency: Once users have learned the design, how quickly can they perform tasks?
  • Memorability: When users return to the design after a period of not using it, how easily can they reestablish proficiency?
  • Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
  • Satisfaction: How pleasant is it to use the design?

Perhaps thinking about each usability principal individually would help to better define your testing methodology.

Your variations of the on-boarding suggests you are most concerned with learnability.

Delaying actual product use after the tutorial as @JeromeR suggested is beginning to approach both memorability and the overall effectiveness of your on-boarding content.

If the testers lack the proper knowledge to use the product, I would personally use an on-boarding process to get them up to speed. If you do not, I think you would be testing much more than the usability of the product interface.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.