I want to compare an existing function in an existing solution (search filters not being seen and/or used on our corporate intranet) with one or two prototypes.

What's the best way to this? How do I compare and measure and verify if something is better/clearer easier to find without telling the test person what I'll be testing? The problem, or problems, are:

  1. They know that they are being tested.
  2. I'm comparing a fully working existing solution with limited prototypes (thus the scope is limited) and hence then know they are tested.
  3. And because of this limitation, they know what I'm testing. So, if they normally don't use the search filters (either because they have never seen them or knew how to use them) they will for sure try to figure it out now in order to complete the task.

So, how would you go about this?


as I understood your main worries is to collect unreal data from running those tests while the employees already know about that, hence do not have the opportunity to go ahead later with the right prototype to develop.

How do you know that this function hasn't been used in past from all the users? Do you have a report with metrics that prove this? If yes I think you can go ahead with no problem.

Your Question: How do I compare and measure and verify if something is better easier to find without telling the test person what I'll be testing?

I would run the test on small teams. You could try to speak with the TL to have a list of up to 5 people to interview (10-15 minutes) with specific open questions about the intranet system used and then move to their desk just to observe their way of use of the system (10-15 minutes).

Do not mention, during the interview or shadowing, which specific features are needed to be improved but inform that involve all the system.

During the shadowing do not help them to complete any tasks but just observe how they use the system.

If everybody knows about the test then I would probably, involve mostly the employees that can interact more with the filtering function in their daily tasks as their use would be more valuable than others. Your aim is to improve that function, therefore, have those as user test is helpful.

To compare past data and actual data then could be enough to understand which solution is the best.


I don't think it's a problem that users know they are being tested. It's often the case in usability testing. You need to find a task that is as close as possible to a real use case where users would benefit from using the filters (without mentioning filters).

Your prototypes need to allow the users to move down 2 or 3 different paths, not only the one using filters.

To avoid the obvious discrepancy in quality with the current website, you could reproduce in low-fidelity prototype the way filters currently work live. It might also limit the impact of familiarity bias.

For a good setup, you should use 3 groups of users with group 1 testing prototype A, then B, then C, group 2, prototype B, A, C, group 3 testing C, B, A. This way, they can compare the different solutions and give you more detailed feedback. Changing the order of the prototypes will allow you to take into account possible priming effects from one prototype compared to another.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.