How to measure an app's usefulness?

Question

That's the main question, but here is some background:

We created an app whose goal is to make raw health data easy to understand for the average person. This health data includes statistics about the number of people in each state who suffer from common diseases/disorders. The target of the app is young adults who are looking to move to a new area, and may be concerned about info like smoking rates, allergy alerts, HIV/AIDS rates, etc.

Now we are channeling our energy towards making this an academic research project. Our main goal is to find out if this app is useful (does it help people understand publicly available health data?).. But how can we measure this objectively?

Answer 1

The answer to your question is not as complicated as you think.

First of all, what is "usefulness"?

"Usefulness concerns the degree to which a product enables a user to achieve his or her goals, and is an assessment of the user's willingness to use the product at all".

From Rubin and Chisnell

So, there you go. Lots of information in that definition. Taste each word and phrase and figure out what that really means. This boils down to two issues, right?

1) Define the criteria for "achieve the goal". That is your decision to make. This could be by asking the user, or it could be by checking if a task were completed, or it could be by setting a maximum time to accomplish a task.

2) Find out if the product meets these criteria. This might sound complicated, but imagine that your building a desk that must be able to carry the weight of 100 kg. The test is simple: Put 100 kg on the desk as see if it still stands. This is the same thing, but with two minor modifications.

• You need a bigger sample. We're talking statistics, and you need to perform more than one test to verify your thesis.
• You might consider some other questions. A bit more sophisticated test. Eg ask if he/she would use it again, or whether he/she would recommend it to a friend. You could even trow in some inverted questions that needs a negative answer to conform the usefulness. Eg "Do you think you would receive more useful information if you asked your doctor". You can confirm the reliability of your data by mixing "yes/no" questions.

Now, by getting loads of qualitative data from these tests, you can start analyzing the data and compare the results with the criteria you defined.

Answer 2

Instead of asking "Does it help people understand publicly available health data?", a better question may be "In what validated scenarios does public health data contain the answer to people's questions/problems/needs?"

The target of the app is young adults who are looking to move to a new area, and may be concerned about info like smoking rates, allergy alerts, HIV/AIDS rates, etc.

...seems like an assumption that needs to be validated. Are young adults concerned with this data when moving into an area? The success of the app rests on the truth of this assumption.

This is a question properly answered by doing user research before creating an app. Here's a few good thoughts from a recent article on UXMatters: How to Know When Your Product is Going to Fail

A primary purpose of UX research is risk mitigation. When we perform research we often look for indicators that let us know whether we are on the right or the wrong track with product development—and, if we’re on the wrong track, how we can get back on the right track.

If you put a product in front of users and they don’t tell you that they want it, you might be on the wrong track.

At the current point in your development cycle, it could it be as simple as: "Usefulness is indicated by usage" - analytics will tell you how useful your app is. If people do not use an app or continue to use it (providing it receives enough promotion/publicity) it is an indication of a flawed user need scenario.

The concept of validating assumptions (and user interest) early is a key idea in The Lean Startup. If your assumptions do not validate, it does not necessarily mean that the entire idea is faulty, but that a pivot may be needed to a different approach that better connects the useful data to actual user needs.