How to define KPI for UX?

Question

I'm in a situation that I have to setup an A\B Testing for the UI of my tool, in order to make a decision about the better UI that my users would prefer.

The tool is a desktop application, which means I can't track the usage by URL links. I have to store the KPI metrics on a file in the users' desktop, and then collect them (I ask for permissions of course - that's not the issue here).

The tool is very colorful and offers a rich UI with many functions.

Well I'm now considering what metrics should I collect? Some of the things I think about:

• Duration of running the tool (each time)
• How many times the tool was launched

What other KPI and metrics should I measure?

Answer 1

A good metric would be something that you could relate to the intended use of the app and unambiguously determine what it means. You want to avoid any metric that has multiple interpretations, because you could easily end up deceiving yourself. I'm assuming you don't have an opportunity to get feedback in the users' own words (?)-- that's the least ambiguous metric.

Ideally you want to measure ease of use and (if possible) enjoyment/satisfaction. The latter can only really be measured by looking at how much / how often the user chooses to use the app; if it's a business app people have to use, you can't really measure satisfaction without asking them. Ease of use should be easier: just measure the time between the user wanting to do something, and that thing being done.

For example, if the task is to fill out an expense report, you can measure the median time from start to finish. Some outlying results will be very long (user was interrupted by a phone call), and others may be very short (user didn't fill out the form properly). The latter might be a useful metric in itself-- you could record, say, the number of times the task took more than one standard deviation less than the mean.

That metric won't be so useful for tasks of unpredictable length (like writing a letter)-- if it takes twice as long in version B, is that because version B is so much fun, or because it's hard to use?

To measure overall satisfaction / engagement / enjoyment, I think the best metric would simply be: does using the application (or part of the application) make you more or less likely to use it again? This might be worth asking about on the Statistics stackexchange, but a crude approach would be to plot, for each launch of the application, Y=(time since last launch) against X=(total hours of use by this user over the past week). If you take the regression line of this plot, its slope should be proportional to how positive the experience is.

Answer 2

I agree with botato that satisfaction is worth measuring. If you want to go that route, you can look into using surveys or other methods to get a net promoter score (NPS) which you can compare to your competitors and/or a baseline (i.e. collect it on a regular basis and monitor trends). NPS isn't perfect, but it is well respected in the business community as a measure of satisfaction and will have more weight for that reason when you present it.

All that being said, satisfaction can't be measured in an A/B test. That being said, it still depends on the context of your application as Esin and bobtato have pointed out. If it is a medical application where the user documents a patient's condition through defined lists, perhaps the best KPI is number of mistakes (lower being better) rather than time to task completion, or at least in addition to time to task completion.

If you do end up measuring task completion time, you can do this in Google Analytics by passing a time stamp through an event when the user "starts" the task and another event with another timestamp when the user finishes. Then you measure the time in between and compare the lengths on the different versions.

You should also consider doing interviews and trying to understand your users' end-goals and emotional goals ala About Face and seeing if you can measure how well you are contributing to those goals. For instance, I work with lots of clerical healthcare users, and they want to feel like they are contributing to the care of the patient, not just being productivity monkeys. Thus, one of my KPIs in addition to net promoter score is a question in a regular survey about how well the application helps them improve patient care. If they think it is doing well, we have created the right emotion. If they don't think it helps them improve patient care, but they do think it makes them more efficient, we might be covering our bases, but not really doing as much as we should be.

Answer 3

If possible you should always try to use a combination of quantitative and qualitative methods. In general an A/B test is all about quantitative measures, but even though "numbers don't lie", they may still be deceiving.

For example, the assumption that shorter execution times leads to better UX might not necessarily hold true. In general it is probably mostly valid to believe that shorter execution times means better usability and usability can in most cases be seen as an enabler for providing a good user experience, but it really cannot stand alone and there are exceptions to the rule.

For example Flappy Birds was horrible from a usability perspective. It was so "unusable" that people could hardly play it. Yet, it succeded to communicate a disturbingly addictive experience (what is "good" UX?) that made people use the app over and over and over...

Same goes for number of launches. Maybe that specific build of the app kept crashing so users had to constantly relaunch it? Or maybe some other factor may have biased the results.

My point is that it is a good idea to set up some quantative KPIs to identify possible problematic areas, but without the ability to survey at least a small sample of the actual users or a test group, you will never know the truth about what people really think of your product.

Set up an A/B test, then try to get access to surveying some of the users about topics your A/B-test suggest might be problematic.

For the A/B test I would focus more on task completion than on execution times and launches. For example you could measure signups, users saving a document, users completing the first level in a game, users purchasing additional services, etc.

Additionally I would look at retention rates. The user just signed up - but does he log back in within XX days?

Answer 4

Where do I start? There are some many components that comprise of the overall user experience that it is really difficult to try and sum it up in a few metrics. But to get you started, here are some ideas to help to narrow down the scope and fill in the gaps:

UX = Usability + Utility + Engagement + Other factors?

• Usability (how easy is it to use) can be measured by a number of standard metrics and heuristics... need references here
• Utility (how useful is the content) can be measured by a number of conversion or business driven goals... need references here
• Engagement (how much users interact with the product/service) can be measured by a number of social and usage statistic... need references here

Just take for example, usability can cover a number of things from inclusiveness of design, which also covers accessibility, so you can see there are many levels of detail you can drill down to.

What about the 'Other factors'? Well, the most common thing people refer to is the aesthetics of the design, which is often subjective but can still be measured by certain qualitative methods. Then there are things like trust and credibility, reliability, security that are a combination of usability, utility and engagement.

Answer 5

Any meaningful usability testing should start with the use case scenarios. You can infer metrics only in that context.

"What is the perceived ease of use of this application?" is a very general question. It does make sense and it should be asked in testing, but it is hardly susceptible to measurements.

"What is the perceived ease of saving a new instance of a file in the application?" is something more specific and easier to analyze and measure respectively.