Conversion rates for captive user systems

Question

In the Real World, conversion rates and bounce rates bundle together factors of frustration, fun, efficiency, and cognitive load in a couple convenient metrics. I work in an enterprise environment in which traditional measures don't apply, like the answers to the questions:

• Did the customer buy something? How much money?
• Did the customer go from trial app to paid?
• Did the user click on an ad?
• Did the visitor create an account?
• Did the visitor sign up for a newsletter?
• Did the visitor ever come back?

Conversion rates directly tie what the user does to major goals software's company, and they're essential for some rigorous interface studies like A/B testing. But I design a captive end user systems for which the usual questions don't work because:

1. They use it, but they hate it. Many of our applications don't have alternatives (e.g., application sites, HR management systems, custom procurement software). If your boss tells you to use it, you will use it.
2. False bounces. Many of our applications get used in mandatory training even if the user isn't in an app's target population.
3. Nothing to buy. Most users aren't in a position to give money to apps that they use.
4. Accounts for all. Rightly, most visitors get accounts the moment they visit the webapp (or before) based on a corporate directory.
5. Small market. Five thousand users give a lot less statistical significance than 50 million.

What are examples of more appropriate metrics for captive end user systems? How can we derive conversion and bounce rates when traditional measures don't apply?

Answer 1

In a B2B environment (in which captive audiences are more common) one of the biggest hurdles is that the buyers are not the users. Purchasing is often done through committee, assessing the needs of the business and then considers the solutions from qualified vendors. Although user considerations are often part of the decision process, prioritization is more often given to the competing interests of budget and existing infrastructure.

There are several ROI metrics that you can focus on to show how focusing on the usability of a product will improve performance, and ultimately the bottom line.

Human Factors International lists out several reasons with associated ROI calculators.

Increased productivity

In a captive environment where a single piece of software has a high usage rate, saving even just a few seconds on a particular task can mean huge gains over a relatively small amount of time.

Reduced reliance on help desks

Both vendors and purchasers hate help desk calls. Vendors have to pay people to answer the phone, and purchasers have to pay people to call in. Reducing the number of calls to the help desk in good for the ROI of both companies.

Reduced costs on formal training

Training in a B2B environment is often times unavoidable. Complex processes, regulation, and standards compliance are just a few examples of why users can't just figure things out as they go. But reducing the amount of time the user has to spend in the classroom means they're out on the floor working.

Reduced learning curve

Related to the above, reducing the learning curve reducing the amount of time the user needs to spend in training. If tasks are more discoverable (process restrictions allowing) and quicker to pick up, the overall initial time to learn the product can be reduced. This brings users up-to-speed more quickly and more productive.

In the case of market size, and finding statistical significance in a smaller user base, Measuring Usability has an article on 10 Benchmarks For User Experience Metrics. Finding issues in B2B software is called out specifically:

9. Usability problems in business software impact about 37% of users: In examining both published and private datasets, we found that the average problem occurrence in things like enterprise accounting and HR software programs impact more than one out of three (1/3) users. While that's bad for a usable experience, it means a small sample size of five users will uncover most usability issues that occur this frequently.

So, while the market may be smaller, the number of issues can be found by a smaller sample size.

The article discussed several other types of metrics that would be useful in a B2B environment. Your ultimate goal continuing to show the ROI of improving the metrics.

6. Average Task Difficulty using the Single Ease Question (SEQ) is 4.8: The SEQ is a single question that has users rate how difficult they found a task on a 7-point scale where 1 = very difficult and 7 = very easy.

7. Average Single Usability Metric (SUM) score is 65%: The SUM is the average of task metrics—completion rates, task-times and task-difficulty ratings. As such, it is impacted by completion rates which are context-dependent (see #1 above) and task times which fluctuate based on the complexity of the task.

The ultimate metric that you are seeking to show is a reduction in the continued cost of use of your product. While the product might cost $100/seat (for example) it might ultimately cost an additional $1000 in person-hours a year because of poor usability. Reducing this number, which you can show in part through the metrics listed above, means your product is more attractive to the buyers and the users!