How to observe users without seeing confidential data

Question

My team's application lets users share files in the cloud. It's a bit like Dropbox, but our product encrypts everything to ensure security and privacy. It's intended for business users who need to let colleagues and clients see sensitive docs (accountants, financial advisors, lawyers, and government).

I don't know enough about how users actually use our product, and I want to see people in action, if possible.

I have been given the names of some friendly customers who would be happy to have me visit their office, or have a phone conversation and maybe a TeamViewer session to see their desktop. I would like to sit and watch them work for an hour or so, to see how Lockbox fits in to their work.

My problem is that these people will be working with confidential material. I am OK with signing an NDA, but I'm not sure how they will all feel about that.

Is there any practical way that I can see how users work "in the wild" without seeing their data?

Answer 1

1. Make them see you as "one of them". I don't mean to deceive - the users will know who you are, and this is a good thing, because research should always include informed consent from human subjects. But try to walk the walk and talk the talk. Adhere to their dress code and not to yours. Build rapport with them, be sensitive to their inner-office culture and try to imitate it. When you are unsure if a behavior is accepted, just hold back and be unobtrusive. Do not do something one of their colleagues wouldn't do (such as whipping out a notebook and scribbling what you see - you will have to rely on memory this time). This will make them perceive you as part of their regular work environment, and also as non-threatening. They will feel relaxed, be more natural in their work (a great plus even when there is no confidential data involved), and not resent your presence. We are taught to hide secrets from dangerous people; make it clear to their instincts that you are not a dangerous person.

2. Give them all the opportunities for graceful retreat they might need - and communicate clearly that you are doing it, and that you consider it their unquestionable right to take them if they feel like it, without the need to justify it. LindaBrammer already mentioned some great ideas how it can be done - give them time to set up, remind them to hide things they don't want you to see before you have the opportunity, etc. You might ask them, before setting up the observation appointment, to do it for a day where they are working with information requiring lower clearance (if this doesn't skew your results).

3. If there are third parties present (e.g. your users' customers), let the user decide how you two will handle that. You might have to step out while they are there, but if she allows you to stay, it is up to her to decide how to explain your presence and how much choice to give them over it.

4. Finally, there is no real way to avoid seeing the confidential information. If you want to see how users work with information and most of this information is confidential, there is no way around it. You just have to be professional about it. Do not record any of it. Of course, much of it will stay in your memory - you will have to put your brain in active recognizing+analyzing+recording mode to ensure you can make conclusions about their workflow. Besides, being in an unusual environment, your brain will notice more details and retain more distinct memories than from a habitual environment. But while you can't unsee what you have seen, you can keep it for yourself. Do not report on it, not even to your boss. Even in pressing situations, if your team is pushing for solution A and you have a good argument why B is the better solution, but your argument is based on the confidential information you saw, do not make the argument, even if this means losing the discussion and your team getting committed to the worse solution. Generally, behave as if you were bound by the same oaths and legal constraints as your users are. If you don't know what they are, look up their professional organisation in your country and read their code of conduct.

An example: I once observed how a physician works with the hospital health care information system during patient appointments. This meant that me and a coworker had to sit in during patient appointments. We had a conversation with the physician before that, and also arrived before the first appointment and let him talk a bit about his work and what kind of patients we are going to see. Then we put on scrubs and sat in the corner. The patients came in one after the other. The doctor just told them "These are young co-workers, here to observe how an appointment is conducted". It was technically true - we did work for the same university hospital, and we were there to observe - and probably a bit deceiving, because the patients were more likely to assume that we are medicine students and not from IT. From the viewpoint of the physician, this did not seem to be a problem, and it let us observe a more genuine workflow.

We just sat there, did nothing visible, did not interfere, did not get up to go closer to the screen even when we would have loved to see exactly what the doctor is doing on the computer but were too far to recognize it. Both the doctor and the patients were very relaxed, they behaved as if we were not there. A man discussed bladder problems, a young woman bared her upper body for a stethoscope examination without batting a lash. When we were done, we hurried to write down everything we had observed and concluded about the interaction between physician and software - but were careful to not make any mention of the interaction between physician and patient. I think we handled the situation well, at least the physician did not indicate seeing any problem, and his superiors (who of course knew of the observation) never had objections or complaints.

If I had been asked before the observation, I would have suggested that I go out before a patient has to disrobe for examination. But it didn't come up then, and during the examination, it came so suddenly we did not have time to react. It was the doctor's and patient's decision to just do it without ever paying attention to us. In hindsight, I think it was less awkward for both doctor and patient to do it that way and just trust us than to start negotiating our behavior. So, you can see that sometimes users' worries are not what you expect, and it can even happen that the solution best for them is far from what we'd have guessed. Let them give the tone, be respectful, and they will be prepared to trust you.

Answer 2

There are well-documented studies involving ethnographic research in hospitals, which can be difficult to gain access to as research settings, because patient records are highly sensitive. But people do get access to them, by building relationships over time and signing NDAs. The agreement you strike with your subjects may include limitations on recording or screen-captures. You can't avoid seeing some data while observing people at work.

But you can avoid taking photos or videos, requesting screen shots, or other artifacts. You can include this in your recruiting conversation, and in your NDA. Here's an example from healthcare, an area with clearly sensitive patient information:

"Pederson and Wolff (2008) documented ethnographic research in two physical therapy clinics in the USA, to understand how small health operations work. They had originally wanted to observe at general health care clinics,... but had problems getting access to observe at these sites. Therefore, physical therapy, in which a lot of patient treatment occurs in a semi-public gym space, seemed like a good compromise." -from Research Methods in Human-Computer Interaction by Jonathan Lazar, Jinjuan Heidi Fent and Harry Hochheiser, p. 239

You are fortunate to have friendly users who are willing to share their workplace with you. I recommend leading with the purpose of your study when you approach them. You want to understand how the tool fits their work, as you say. There is no substitute for observing real users in action.

Be mindful of privacy, and give your subjects time to set up their workspace for observation. For example, before beginning the screen-sharing portion of a research call, you can say something like "before we share screens, we should both close out of anything that needs to stay confidential." Start sharing when they indicate they are ready.

When you are observing in person, sit far enough away so that you get a general sense of the content, but can't see details too clearly. They will invite you to lean in to see details of things that are important to your understanding.

If the material is highly confidential and you shouldn't see any of it, there are filters that some people in fishbowl offices use that make it impossible to see their screen while walking past their office.

Answer 3

There are couple of things you can do from engineering side if you are delivering applications (like windows. mac, android clients) for various platforms. Engineering team can modify the product to gather telemetry which depicts the user behavior. For example, if your application has some controls like buttons, text selectors, graphics etc.. then on each click or action on them can trigger telemetry which will be sent to your servers. Then you can build dashboards using the data. The data can be encrypted before sending to your servers so that your clients will have a good sleep :).

Answer 4

There are a couple of avenues to try. Certainly, an NDA is something you could respect, especially if you are being invited to "come see how we use your software". It sounds like your client is comfortable with it, even if you aren't. But direct observation would be my first choice.

You could set up a camera at a distance far enough from the screen to not resolve individual characters, and observe usage that way. You'll want to set the cursor to something large and high contrast for the test so the camera can pick it up. You could offer to let the client screen the video to assure themselves there is no confidential information visible before providing it to you.

Similarly, web analytic tools like CrazyEgg can track mouse movements without recording specifics of data. If it's a web application, it's a matter of adding a script call (and paying them some money.)

You can often record the screen flow by name, instead of capturing an entire screenshot or collecting the exact data being entered.

You can ask to place a camera on the screen facing the user (the sort built into every laptop these days.) Record the user's expressions during the test: concentration, puzzlement, exasperation, boredom, etc. Correlate the timing with the screens being viewed: how long did they stare at a screen prompting them to "Enter description of incident, using fewer than 40 upper case letters and no alliteration"?

I consider raw timing to be one of the best indicators of usability for a business product. If the users have to stop and think, it's wasting their time. Once I find the bottlenecks, I look at the screens: is the task too hard or complex? Is the wording of the prompt too confusing? Lengthy? Is it redundant? Cluttered? Stark? If the users have to re-enter the same invoice number on seven screens, it's a waste of their time. If the users are frequently making typos that force them back to re-enter the data, the data may be too hard to enter. You might not learn the specifics of the fault, but you can quickly locate it. And timing usually reveals nothing about the choices they made, the identity of the clients, or the nature of the data.

You can also consider setting up a usability lab, and provide your testers with sample sanitized data to be entered. Hold the test away from your customer's premises so as not to make the client uncomfortable that you'll see something they don't approve of.

Answer 5

I face the same problem in my research too because the app involves confidential and personal data. Our solution was to get participants to use the demo version of the app. I would provide them with made-up data and got them to perform typical tasks, in various scenarios. This yielded a lot of insights and in fact, it was a lot easier to observe and record their workflows on the screen. Participants also felt less self-conscious when they weren't observed in person because this sort of tests always makes people feel like their abilities are being tested, no matter how you frame it.

I didn't record their faces because the information wasn't needed. It would be another layer of privacy and consent issues and we were already recording what they were saying.

It seems nice and ideal to be able to observe users like they do in user experience research case studies but it's not always possible and may not be necessary for the research topic.