How to do a mobile field test?

Question

The Smartphone have become a part of our daily life. Users are always online and use their Smartphone in very different environments. Smartphones are sometimes used in a calm, not distracting environment while the user is watching TV with half an eye. Other times the user is standing in line in a drugstore updating their facebook status. Next, they might reply on an e-mail while on a shaking, crowded train or exercising with their smartphone handling the training app.

How do you do a mobile field test in these different environments with different distraction to get a useful result?

Answer 1

If you can be physically present while the user performs his tasks, mainly focused on behavior:

• fly-on-the-wall technique ('shadowing'): simply observe, no interaction, no questions during the task (afterwards is fine). Allows you to see real-time behavior in a natural setting.
• contextual inquiry: a situated method where you ask questions while the user is performing a task in its natural context. It's a bit more intrusive than pure observation, but can lead to fruitful conversations.

If you need more insight into people's thoughts, emotions and subjective experiences rather than their behavior:

• experience sampling: at regular intervals, or in response to certain events, you ask your participants for a reaction (such as describing what they're doing, how they're feeling, filling out a questionnaire, or taking a picture).
• diary study: participants keep a diary recording their activities and experience. The main difference with experience sampling is that with diary studies the participants can choose the moments when they want to work on their diaries

For specific questions or hypotheses it can be wise to work outside the natural settings. For example if you need to find out the error rates caused by working on a moving vehicle, put your participants on a moving subway. If you expect there could be much distraction during use, make sure you distract your users during using testing by having them perform a secondary task simultaneously. Or simulate an extremely high event rate.

Answer 2

The best way is to "shadow" someone. Especially with mobile phones and habits, people will quickly forget someone is observing them.

Here is an article on the subject that goes over it briefly:

Stalking user intercom article

Make sure you write everything down, don't ask them questions until they completed their task. Pay them for their time. Get an agreement signed. I believe you can stream their screen, which would be very useful too.