Instructional overlays (feature tour) for formative usability tests?

Question

I'm a UXer in a little software agency and we are currently redesigning a large and quite complex software with lots of features etc. By now we have developed an MVP that covers a part of the entire software. So, currently, users can create new emergencies and search for bus and hotel suppliers.

Premise: the goal of our usability test is utilitarian and not experiential.

In our first usability test of our MVP, as expected, I've noticed that users struggled with the new interface and didn't even notice the new features that we purposefully designed to solve some pain points they had in their old software. At the end of the test we told them about these features and they actually found them useful to overcome those pain points. Besides, as the users were completing the tasks they were given they started to get more familiar with the interface and struggled less with the interface.

Since most complex software do have instructional overlays and such for new users, I was wondering if it would make sense to instruct users before they take the usability test or not. I'm asking this question because I'm having a hard time at classifying usability problems that rise from the tests.. Indeed some of these problems are due to the fact that the interface is completely new to them.

What do you suggest?

Is there any literature that deals with this problem? My best guess is that by instructing users with overlay marks we can get the best out of our usability test.

Answer 1

If you want to test usability of your instructional overlays (or onboarding), then yes, you should include those in your testable MVP. If you want to use instructional overlays just to pump up the test score, then no. That is not how usability testing works.

You conduct usability testing to find usability problems in your system. Your current results clearly show that your users "don't get" the interface of your system or interactions within. With instructional overlays you are just giving your users the answers before they even take the test.

If users don not find or notice the new features, then your system is not usable. No matter how useful those features are, if users do not find them or do not know how to use them, your system is not usable.

Besides, as the users were completing the tasks they were given they started to get more familiar with the interface and struggled less with the interface.

It seems your system has a good learnability but that is not enough. What you probably need to look into is how findable and recognizable things are.

Answer 2

That's a very good question. You want your tests to be as realistic as possible. And you say that in real life there will be some instructional overlays in place. However, they will probably be displayed only during onboarding, so the real question is how learnable these features are - will the onboarding suffice to teach users about the new features. That is something that can't really be tested using regular usability testing, because learnability is directly affected by the frequency of usage, and you'd need to imitate that for your conclusions to be applicable.

Assuming you can't imitate real usage over time, I would try to guesstimate the usage you expect. If these features are meant to be frequently used (based on actual frequent user needs), you can assume that after initial onboarding they will be learned quickly - and then test them with the overlays, to establish whether the overlays are enough to get them through onboarding, from which point it should be relatively smooth sailing.

If the features are expected to be rarely used, you can't really rely on this learning curve, and then I would test them without the instructional overlays because they'd need to be intuitive enough on their own.