Objectively evaluating competencies of a UX Designer

Question

My company has recently started an initiative for objective quarterly evaluation of employee performance based on the Rockefeller Habit's "One Page Strategy Plan". One of the key sections within the plan are measure-able targets for teams & individuals.

For example:

• Sales Person: a target of converting 100 leads into new customers in a quarter.
• Support Staff: a target of 85% satisfaction rating on customer tickets

Our Product team is struggling to find concrete metrics for evaluating performance.

Are there any recommended metrics for evaluating competencies of a UX-focused Product team and for UX Designers?

A target like shipping X number of features a quarter does not focus on whether we're providing a sound solution to the customer problem, or whether we're even tackling an important customer problem. A customer satisfaction rating for a new feature can be subjected to response bias. A measurement of minimum X concept/user tests for new work, means we always have user feedback. However, we're missing the other side of how effectively we're utilizing the feedback.

We've also considered things like NPS and customer churn reduction. However, so many factors influence these items that it's not fair for the team to be basing performance of them.

I'm curious as to what experiences people have when it comes to performance reviews. Or is it truly a pipe dream for UX Designers to know whether we're doing a good enough job.

Answer 1

Great question although I pretty much go along with Khoi Vinh's thoughts that performance reviews for designers can be a ghastly affair and of little real use.

As he points out:

The New York Times, to its credit, was a place where designers received reviews from people who actually understood our work, who were conversant in the vocabulary of design and who comprehended the value we tried to bring to the business. That may not be uncommon, but it’s also certainly not the rule for the profession, especially for those who work in-house at companies whose business is not principally design.

I don't think any single measurement is going to give you want you want, but there might be a selection of quantitative and qualitative KPIs and metrics that are relevant to your organisation that could be 'factored' in some way to give a usable result.

CXPartners has a Big List of KPIs and metrics. It's a seriously good resource! Perhaps it will provide some inspiration.

But there's no real substitute for real reviews from real people who know what you do - back to Khoi Vinh.

Getting a review from someone who has an inaccurate understanding of what you do is a waste of everyone’s time. Unfortunately, yearly salary increases are often tied into these reviews, further inflating the value of bad data.

And with that in mind, perhaps the use of 360 degree feedback could be utilised (in conjunction with traditional appraisals) somehow as a larger component of an accumulated metric.

---

Knowledge disclaimer: I have no direct personal experience of any of this. I merely wanted to provide food for thought. I'd be surprised if someone came up with an answer recommending a system that works, but would love to see it if there was one!

Answer 2

As much as I hate trying to "quantify" success in areas of programming or UX, there are some possible key performance indicators that have been suggested:

From Designmodo:

Quantitative UX KPIs

Task success rate
Time on task
Use of search vs. navigation
User error rate
System Usability Scale (SUS)

Qualitative UX KPIs

Reported expectations and performance
Overall satisfaction

From uxmatters:

Answer 3

The problem with nearly all performance review systems is they mainly track one's ability to measure up to the performance review system rather than measuring actual work.

The television series The Wire summed this up succinctly with the phrase juking the stats.

The problem with measuring objective output of the UX team is that the output of the UX team is rarely the actual product output. In larger companies, the product (often software) will have a UX that is only partially controlled by the UX team. You have management, IT, development, consultants, outsourcing firms, etc all getting their hands on it and a lot of the final UX will come down to all those parties--not just the UX team.

Answer 4

I would say you would use AB testing to determine the outcome of the UX work being done.

From my point of view, the only way to measure UX changes is to measure how the new UX performs against the old UX, and can only be done objectively with AB testing.

I would measure Funnel completeness and conversion. At the end, the main objective of UX is to increase conversion.