User Experience Research on Users Total Digital Environment?

Question

When we do User Experience Research we test a single application or even a single feature of an application. But there are no users who only uses one application. Users today are using several different applications on several different platforms in work, while traveling and at home.

This could be a Microsoft environment at work using Windows 7, SharePoint intranet and the Office applications. On top of that there could be a Time Registration System, an Enterprise Resource Planning system and a Quality system. This could also be accompanied by a Smartphone and all of its accompanied applications.

At home the user might have a tablet and a private Smartphone different from that of work. Then the user have credit cards, bank cards, gas station cards and all other cards where the user needs to remember the unique PIN-code on each card, which might change when we get a new card. Not to mention all the username/password combinations the user have to remember and change every once in a while depending on IT settings.

This is all part of the digital environment the users of today faces. I've tried to find research that covers the total digital environment of users, how they deal with these environments altogether. So far I've been unsuccessful finding relevant research. Have this kind of research been made?

Answer 1

Digital living labs are an option, stuff like Digital Spaces Living Lab andAn internationally distributed ubiquitous living lab innovation platform for digital ecosystem research. Living labs are a sort of holy grail, but they're also extremely difficult to set up relative to simple tests.

Instead of hoping for a pre-baked solution, look to examples of psychological research "in the wild". You might consider a briefer form of Longitudinal Study where you give your users an app to test and either ask them to use it naturally or ask them to use it every so often and collect data as time goes on.

There's some good ways a Longitudinal study can gather information without being over the user's shoulder 24/7:

• Intermittent surveys or diary-style reports from participants
• Automatic data collection tools (used with permission)
• One on one or focus group interviews before/after/during the study

Ideally you'd gather feedback from a wide variety of users who all naturally use the app, but there's already going to be plenty of selection bias, so it might be worth asking participants to just use the app every X days or for an hour every so often since you're probably dealing with a limited pool of participants in a limited time frame; if you're not conducting peer reviewed research that is intended to be rigorous and reproducible, it's more important that you get some data out of people than that you run a perfectly valid study. You want to be as valid as you can, but optimizing for validity and reliability is very difficult to begin with in these tasks. Keep the data as real as possible, but be practical.

Instead of trying to record every second of the user's life with the product, collect intermittent surveys of their interaction with it. It's good to have a mix of some numeric measures like Likert scales along with some direct, relevant questions and a free-response section. This allows you to measure long-term trends so you can see the apparent difficulty/quality of experience curve of the app while also gathering free response insights which don't fit neatly into your stock questions.

It's also ideal that your app itself collect some tracking data of the users; gain consent and fully disclose what data the app will collect and when/how you'll get the data from the app. Ideally you could monitor usage of related apps, but this is rarely practical. It's more likely you'll be able to produce a special test user version of your product which does all this tracking. The survey is there to help fill the gaps.

Finally a one-on-one session or focus group with the users before debriefing can help you gather as much insights as possible (don't say you had anything to do with designing the product!). Not everyone fills out free response sections (or gives useful information in them), talking to them personally and casually can help glean as many insights as you can.

Obviously much more rigorous than your standard "bring 40 people in and have them click through a few screens" usability test, but this doesn't have to be as hard as it initially sounds. Collecting analytic data is pretty common in apps today already, and drafting up some surveys won't kill you. Remember your longitudinal study doesn't have to last years, pick a reasonable length considering the learning curve you want to study (new user to proficient? New user to expert?) and project time constraints.

Answer 2

These are called living lab methods, imagine them like a scientific version of Big Brother.

Google has thrown me this Living Lab Method as first result, but I've read about them in the Interaction Design book.

On a sidenote, I wouldn't be surprised if real Big Brother or other live-together reality shows would be used for this (as part of a silent sponsorship), I mean, the cameras are there, it's not quite a natural environment, but a living lab always has deficiencies.

Or you can install it at all the phones and workstations at a company, but I'd be terrified a bit on this. Living lab is easy. You sign a paper, you get benefits, and you're watched, that's clean. I wouldn't participate in it but there are people who would do.

Beware of the privacy issues: always be clear about what you watch and for what purpose to all those you're watching. Perhaps release your findings towards them at the end of the experiment. People are sensitive about their private life.