Best method for measuring the user's understanding of a pilot app?

Question

The company I work for created a new app that collects metrics on the user's driving behavior. We are running a pilot test of the app with internal employees before launching. What would be the best method for collecting feedback on the app? An online survey... 1:1 moderated interviews...? I think our biggest concern is proving whether or not users understand how to use the app.

Answer 1

The simplest method is to define the end goals that you expect users to accomplish.

Using stackexchange as an example, a goal might be: "did the user search for and find an answer to their question?"

So you'd look at the following: What % of users performed a search? What % of those users clicked on a search result? How many search results did they click before exiting? Added bonus: on exit, ask the question "did you find what you were looking for?"

If high % did those tasks, you can assume the thing is working the way you thought it would. If you see that nobody searches, or that when they search they don't click anything, or they have to click 1000 things before exiting, you can start to dig deeper in with surveys etc about specific actions.

Answer 2

As a complimentary method to the @user5482 answer.

Conduct diary studies

A diary study is a research method used to collect qualitative data about user behaviors, activities, and experiences over time. In a diary study, data is self-reported by participants longitudinally — that is, over an extended period of time that can range from a few days to even a month or longer. During the defined reporting period, study participants are asked to keep a diary and log specific information about activities being studied. To help participants remember to fill in their diary, sometimes they are periodically prompted (for example, through a notification received daily or at select times during the day). (Source)

Advantages:

• elicits qualitative data over different time periods
• reporting events and experiences in context

Disadvantages:

• participant commitment - often times participants forget to input data for the study. This can be mitigated by using reminders to log some data.
• Training or briefing sessions - researchers have to ensure that the users understand what data they have to input, where, how, and when they have to do it.

Here is a great article on diary studies by Carine Lallemand.

Answer 3

You definitely want to user one-on-one interviews for this. Data can only tell you the what but talking to people will tell you the why.

I would structure these interviews as about 60 minutes each. The first 30 minutes I would keep it casual and aim to get a picture of WHO this person is -- what's their general lifestyle, attitude toward technology, what's the job and their daily routine? You want to kind of get a day in the life so you have context about how the rest of the lifestyle may impact their attitudes and behaviors with this app.

In the second half of the session, I would have specific tasks that you want the person to do on the app. Create a discussion guide for the interview so you know exactly what you're going to ask people. Do these unguided tasks as well as some guided discussion -- meaning take them to say the homescreen and ask questions like "what do you think this app is about", "what stands out to you as you scroll on this screen", "based on what you see so far, would you trust this app" -- etc etc.

I'm a UX researcher so I could go into a lot more detail, but that's a good high level.

Answer 4

For getting a basic understanding of if users can figure out how to use an app I like to do unmoderated user tests. That can be done by giving the user a goal (will vary based on how your app works), and see if the user can figure it out. Have the user screen record or since they're internal you can just watch over their shoulder. The purpose of this is you can see where the user makes mistakes. If every user follows the wrong path or clicks some incorrect button you know that it is unclear at that point and needs to be worked on. With unmoderated tests you almost always find something you would not have even thought would be a problem to ask about in a moderated test (in my experience).

Answer 5

I would use 1:1 interview, because you could more focused with them such as their mimics, eyes, gestures ETC and the important one, you will know exactly whether they really using the app or not.

Answer 6

To follow a pilot program you can improve your logs to follow the usage of the application.

With carefully thought logs you can almost see how the person is using the app. Take a sample of few users and analyse their logs very carefully. Take samples and counts of access's to some operations you are most interested.

Of course surveys help, but your test sample may not be qualified to give helpful evaluations for a number of reasons. They may not be as dumb as some users. They know what the app is for. They know they are being monitored and have to access the app.

Answer 7

The best way would be to see how your users are using your mobile app without their knowing it, so that you are able to capture the first hand experience of issues while using the app and also get to know user navigation pattern.

There are user experience tools available for mobile apps which you can try out which allows you to see how your users are using your app and where they are getting stuck.