Prioritize usability problems

Question

There are a variety of references and approaches to prioritizing usability problems, but the I have not identified any single approach that captures all the factors I feel are necessary to include in evaluating severity.

According to Jakob Nielsen, a severity rating should be determined by factoring the following:

• Frequency - Is it common or rare? Does it occur on a red route?
• Impact - Will it be easy or difficult for users to overcome?
• Persistency - One-time problem a user can overcome once they know about it or will users repeatedly be bothered by the problem?

Other sources approach severity by measuring the three factors outlined by the ISO definition of usability which include:

• Efficiency - Can users still complete tasks, goals? Can they do what they want to do? The resources expended in relation to the accuracy and completeness of goals achieved.
• Effectiveness - How much effort is added per the problem? The accuracy and completeness with which specified uses can achieve specified goals.
• Satisfaction - What do the users think about the products ease of use? The comfort and acceptability of the system.

Additionally, neither severity score factors listed above account for the impact the problem has to the business, e.g., direct impact to revenue, brand equity, profits, perceived value, etc.

Last, I have not seen any factors include economics, i.e., how much effort (cost) will it take to resolve the issue. Fixing a medium severity problem with marginal effort compared to a sever problem that's extremely costly can make good sense.

So I'm curious, has anyone implemented a process for evaluating and prioritizing usability problems that include all the above factors (or perhaps ones I'm not thinking) and if so, would you care to share your story?

Factors to be included;

• Frequency
• Impact
• Persistency
• Efficiency
• Effectiveness
• Satisfaction
• Business profit/revenue
• Brand equity
• Cost to fix

Answer 1

On our team, we evaluate the user-based factors, brainstorm a couple of possible solutions, estimate the cost through an Agile development process (tech review and estimation meetings,) then present the viable solutions and their cost to the business decision-makers.

I know this will grate on some people, but I do think it's possible to overanalyze these things. I find that more granularity in analysis doesn't necessarily mean a better analysis. Since the analysis itself costs money, there are diminishing returns at some point.

Generally, I think of it as two core questions in comparison with other tickets/issues: how badly does it affect users, and how badly will it affect budgeting (of both time and money.) The specifics of what, exactly, you use to analyze each of those varies from project to project. (Which is probably why you get so many different opinions on it.)

Answer 2

I occasionally find it useful to place issues against these axis:

^{download bmml source – Wireframes created with Balsamiq Mockups}

Which generally leads to prioritising fixes from top-left to bottom-right.