3

I work in enterprise UX and need to measure metrics for how the usability of our future application will have improved work performance among users. The system the users use today does not record a time stamp and has unreliable metrics. The users have already been surveyed and interviewed for qualitative things like attitudes and job satisfaction, but I need a reliable way to measure quantitative metrics, like time. With the way things are set up now it would involve me literally following users around and timing them as they 'pick' things off of a shelf and scan items into the application.

With that being said, would my presence alter how they are timed? Is there a way to avoid conversation with the user as to not alter the metrics without making the user anxious or pressured? Should I to a 'mock' timing session for a few hours and follow them around until I feel they have passed a threshold where they are performing normally?

  • might it be easier to observe them on CCTV or a set of cameras instead? – Midas Apr 29 '16 at 15:22
  • I think this approach may be less intrusive and I feel really silly for not thinking of it before. Thank you! – Jessica Bruner May 2 '16 at 14:07
1

Would my presence alter how they are timed?

Yes, it would. It's known as the Hawthorne Effect, where subjects change their behavior simply because they are being observed. Measuring Usability has another writeup biases that also mentioned this effect: 9 Biases In Usability Testing.

It's not a matter of you necessarily being in the room. The fact that they know they're part of a usability test will alter the results.

Is there a way to avoid conversation with the user as to not alter the metrics without making the user anxious or pressured?

Yeah... don't talk to them. You don't have to be in the same room to observer them, assuming you have the right room configuration and/or video equipment.

If you are in the same room, your only conversation with them can be to provide instructions and tell them when to start. If you're not needing to prompt the user for any additional information (e.g., "please remember to think out load", or "what are you getting hung up right now?") then there is no reason to speak again.

If they ask you a question (they probably will) you can simply tell them you are unable to help them complete their task, and to please continue.

As long as they know they're being watched their behavior will change to some degree. Any action you take will cause a shift in their behavior. Simple note taking, for example, can become more self-conscious when they notice notes being taken during the test.

Should I to a 'mock' timing session for a few hours and follow them around until I feel they have passed a threshold where they are performing normally?

At what point does one stop caring they're being observed? I'm not aware of a study that gives suggestions on this, and it is highly likely to be very subjective.

In that you plan to "follow them around" - the environment certainly plays a large role in this too. If after (say...) 2 hours you start collecting "real" data but the user is suddenly in a high stress situation, was it your presence or the situation that caused a shift in their performance? Maybe they stepped into an area was a not as well lit as before, which could effect their interaction with the device.

How do you solve it?

You accept that your presence will effect the outcome. There is no way around it.

If you're performing time trials keep every other variable the same, as best you can. Ask them to perform the task with the old system and the new system in a sterile environment with you standing 5 feet away in both situations.

In the case where the activity can't be reproduced in a closed environment, attempt to control the environment as best you can during the activity. Don't just follow them around until they reach that activity - set it up so you control as many of the performance impacts as possible.

If you want to perform a field test of your application, don't expect the environmental variables to remain constant. They will change constantly.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.