Phrasing of diary study prompts to get more user insight

Question

I am in the process of designing a diary study to investigate the interaction between a current software application and its users. I am seeking open-ended data and insights from users that I might not otherwise get from a survey. The users are free to put anything they want in the diary, but to also provide some direction the diary includes daily prompts to the user.

I'm having a little trouble designing some of the prompts used during the diary study, however. I have questions like:

• Describe something in the application that makes your work day easier
• Show a screen shot of something that frustrates you

I'm sort of winging these questions, however, and I was wondering if there is any research out there that investigates what types of questions yield the best results for a diary study, or if there are any existing case studies I can use as a model for how I should be creating prompts within the diary.

Answer 1

A couple methods my lab colleagues used in their 'diary' studies:

• Sending sms during the day to remind the test subject to take a note. Can be contextual (time based reminders) or just generic prompts.

• When the subject is signing up for the study, you can negotiate a time when you can daily/weekly/up-to-you: a) call them up and ask questions regarding their experience b) go and meet them in person to get feedback (and maybe ask them to repeat their annoyance in front of you)

• Make it as easy as possible to submit errors/problems. Maybe you can just ask them to press a button to note the time and the error which the application gave. Maybe they can use record voice notes rather than typing, all depending on the setup.

• This one is a generic user feedback suggestion: Always ask open-ended questions. Let the user describe in their own words and feelings how their interaction was. Do not (try to minimize) asking prodding questions. Like, when they sign-up, you do not need to tell them what details they need to focus on, just explain what the software does and tell them how to record the feedback. Many times users end up digging up problems which you were not even aware of (d'oh!).

Diary study is not about numbers and metrics, it is about qualitative feedback. You are trying to build a sense of empathy with the user while allowing them to work in their own environment without any meddling.

I do not have access to ACM right now, but, I found a couple papers which might be interesting for you to read:

• Carter, S. and Mankoff, J. 2005. When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Portland, Oregon, USA, April 02 – 07, 2005). CHI ‘05. ACM, New York, NY, 899-908. Useful for their discussion of using media other than paper, with references to examples.

• This one is useful for their method of having people use text messaging to send diary entries: Sohn, T., Li, K. A., Griswold, W. G., and Hollan, J. D. 2008. A diary study of mobile information needs. InProceeding of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy, April 05 – 10, 2008). CHI ‘08. ACM, New York, NY, 433-442.

• http://courses.ischool.berkeley.edu/i214/f08/diary.pdf

Answer 2

We've tried many, many wordings, but in the end, these are the ones that work best for finding things to improve in the UX:

• "Help us improve, tell us what we should change to make this (app/screen/option/process) more useful and easy to use."

and

• "Help us improve: what problems did you have with this (app/screen/option) that prevented you from (buying/selling/whatever activity the thing is intended to).

These kinds of questions are always in the context of use, for example a screen or a flow, since users are much more aware of what they are doing at the moment than when you ask them at a later time.

No matter which one we use, some users are naturally inclined to commenting on their problems, while others write about what they think they need, and we have to transform that into something meaningful to the project.

Nonetheless, with enough user comments (50-100, no more than 300 are usually required), you should start to see the patterns emerge.

In the end, you'll see that if questions are open enough (your examples are a bit narrow yet), users will tell you what you need to learn, just don't get too much bogged down at wording your questions, since it won't make a big difference.

Answer 3

It is probably not going to help you. But I am not losing the occasion to present you this :

As you are willing to do Facebook wanted to have a little bit more feedback on the use of their application by their internal users

Facebook forcibly updates employees to the most recent beta version of apps like Facebook For Android and Facebook Messenger. If they run across a problem in one of the Android (or iOS) apps, they can take advantage of a bug-reporting feature Facebook builds into its internal betas.

It’s called “Rage Shake” and the name is spot-on. Employees just violently shake their phone and it automatically logs its current state and sends details to Facebook’s mobile bug-squashers.

Original article

Good luck for your study.