When testing a prototype, do you inform your users that everything is not functional?

Question

There is a bit of a split on this topic within my team. We use usertesting.com and when testing our prototypes we have opted to not notify the users that they are testing a high-fidelity (flexible, polished UI) prototype hence not every button or interaction is functional.

The UX lead has said that we shouldn't tell users as it will set up a bias and encourage our users not to explore much as they will think that any mis-click or button might not be functional. We can always gain insights as to where the user is trying to click.

I can't disagree with this logic but I have seen other tests and an article stating that it is best to tell the users that they are testing a prototype. I can also see frustration when a user tries continuously to click on something that wasn't 'built' in.

Any best practices that some stick by?

Answer 1

If you don't want to immediately make the user aware that they're not using a finished product, then you should definitely make them aware when they go to click on a button (for example) and it isn't yet functional.

I would strongly suggest that you pop up a tool tip or another form of (non-obstructive) message that informs the user that the feature they are trying to access is currently in prototypal development and should be working at a later date.

This stops users from feeling that your application is poorly made, because who on earth would release a final product with buttons that are broken? This also prevents wasting a user's time if they feel the need to report the feature as non-functional, because they will already know it's under development.

Edit: You shouldn't say that the entire application is a prototype - rather, just that the feature they're using is under construction. This will mitigate any sweeping judgement on the application as a whole because it partitions the user's expectations.

Answer 2

This comes down to how you conduct your testing.

Personally, I always inform my test users that they will be working with a prototype so they cannot 'break' the system and will not destroy any data. This makes them feel a little privileged, helps to set them at ease and allows them to move with a little more confidence. It also means that they don't end up with me saying "sorry, that doesn't work yet" over and over.

I also carefully script my tasks so that, if the design is successful, the test user should not encounter any non-functional parts/controls. You can consider the user hitting any non-functional part of your prototype as a design fail - either of the product or the test.

I find that, if you hide the fact that they're working with a prototype, users feel that the product is buggy and that in itself becomes a distraction that can colour Kano, SUS and NPS scores.