2

The NPS score, despite it's simplicity, appears to be a very good indicator of people's attitude towards a product or service (at least according to the marketing department). The value of encapsulating all the factors into a metric is that you can spend more time monitoring and tracking changes, which is the main benefit of a metric (to 'measure' and 'improve') rather than trying to work out exactly what the underlying factors are (which you should do by more in-depth research).

My theory is that you can also encapsulate an organisation's level of UX maturity by the simple metric of how much time a typical UX designer would spend with the user compared to the business stakeholders, and that you could simply express this as a ratio, with the higher ratio reflecting a greater level of maturity (i.e. more time with users compared to other stakeholders).

Based on your experience, do you think this is too simplistic to be applicable, at least compared to more complicated measures of UX maturity models (of which there are only a limited number of questions on UXSE).

Personally I think this is a reasonably good approximation that will give a suitable ball-park figure, but I would be keen to see if there are examples of companies with high levels of UX maturity where this rule of thumb does not hold true.

  • Who is the customer of this UX maturity metric/score product? Is it something for UX managers or product VPs to chart the progress made through UX efforts? I don't think your ratio is too simplistic at all, but I do think it would be problematic to try and track...it's hard enough to get people to track their time even when they're paid hourly! Plus, wouldn't customer service teams kick everyone's butt with this metric? They usually inform/consult on project teams with the business owners. – Luke Smith Apr 30 '18 at 20:36
  • @LukeSmith I mentioned in the question that it would be measured based on the amount of time the UX designers spend with the users compared to the business stakeholders. The consumer of this information would be anyone who wanted to benchmark the organization's ux maturity against other organizations. – Michael Lai Apr 30 '18 at 23:39
  • Thanks @Michael Lai I understand what the metric is composed of, I just don't understand where your data would come from. Would it be self-reported? Manager review of time sheets? Wild guess/gut feeling? Feels problematic to wrap a KPI around. I know you mentioned you're just going for a reasonably good approximation/ballpark, although that gets more subjective (and of course even harder to benchmark against other organizations.) – Luke Smith May 1 '18 at 11:07
  • @LukeSmith since this is a self-reported measure, you can simply look at the time spent on a UX project on actual time with end users versus the business stakeholders. I think this can be done reasonably accurately, or at least accurate enough to know whether you were spending more time, about the same time or less time with the users compared to the business stakeholders. – Michael Lai May 2 '18 at 5:53
1

Creating a metric

You want to confirm whether a metric you come up with (ratio of designer-user, designer-stakeholder interaction) is an accurate measure or approximation of UX maturity.

Define UX maturity

Figure out how to accurately measure UX maturity with an established and trusted method. To be honest, I don't know of any. MeasuringU seem to be creating one, and of course NNGroup has their definition of maturity stages.

Essentially, define what you mean by UX maturity, and figure out how to measure it properly even if its with an interview or long form survey.

Measure maturity and gather your own metric's data

Use your metric to survey designers, and also use the established method (interview or survey) with the same group.

Check for correlation

Determine the correlation coefficient of your metric vs output of the established method (like the maturity stage), and see how closely they correlate. If they highly correlate then your metric may be a good approximation of more complicated methods.

All this being said, I've never had to come up with my own metric before or validate one, but I've come to this solution because whenever I've read about usability or satisfaction measures, they all talk about how they correlate to other established measures. So it makes sense to me that the same process applies here. For example...

"Be aware that the SUS correlates strongly with a much simpler metric, the single-question Net Promoter Score. They do provide different data, but for many organizations, the NPS may be more useful overall, as it’s a simpler metric to collect (one question versus SUS’s 10), and is a well-established general bellwether for the company (even if it’s not as sensitive to UX-focused concerns)." https://www.nngroup.com/articles/measuring-perceived-usability/

Drilling down beyond a single measure

"...encapsulating all the factors into a metric [...] which is the main benefit of a metric (to 'measure' and 'improve') rather than trying to work out exactly what the underlying factors are..."

You're right that you need to do further research to find the underlying reason for a change in rating, but you can also ask higher quality questions to narrow it down.

A great tool I've discovered for tracking user satisfaction is this Happiness Tracking Survey from Google researchers.

The relevance here is how they use one single mandatory question (that asks you to rate your satisfaction), and then drill down a little bit by asking you to rate properties of the application (like speed, reliability etc), then drill down once again by asking about specific tasks undertaken in the application.

Do I think the metric would work?

My hypothesis is no. What if there are an insufficient number of UX designers to properly be across the work being done by a company. I may have a lot of contact with users (performing research, conducting testing sessions etc) - but user-centric design practices are not sufficiently adopted throughout the company, which to me would be a low rating of UX maturity.

  • +1 It seems like most people find it too complexity to try and benchmark UX maturity within an organization because there are too many factors involved. Ultimately the NPS score boils down to the idea that if someone would be happy to recommend your product then it is an indicator that they are satisfied with your product/service. My hypothesis is that if UX designers spend more time with users compared to the rest of the organization, it encapsulates factors that contribute to better quality design. Of course this is based on my own experience hence the question. – Michael Lai Apr 30 '18 at 23:43
  • 1
    Yep, and what do you think of my view that it misses a dimension (last paragraph). You can still try and test it regardless like I suggested. – Zenon May 1 '18 at 1:41
  • You are correct in that the metric will not cover all relevant dimensions, but I would assume that if you are spending a lot of time with users (officially, not out of your own time) then there are enough adoption of user centric design practices throughout the company. Another thing I thought of is that how much time you spend isn't necessarily reflective of the quality of the time you are spending with them, so it is by no means perfect but something that is a double-edged sword like the NPS score. – Michael Lai May 2 '18 at 5:59
1

I'd like to know more about who needs this metric and what decisions they're making on the basis of it. If there are no such people or decisions, is the metric important to even track and report on? Particularly if it defies objective evaluation.

It's hard enough to explain to people outside this domain what UX even is, let alone UX maturity.

You make some good points in this post, or at least what you mention there resonates with me personally. I think in a "UX mature" organization, the difference between designer and UX designer becomes less meaningful:

...I would assume that in a completely user-centred organisation there would be no need to make this distinction, but I am interested to know if this is actually the case.

It is a highly subjective topic of discussion but I observe that some really great, focused product teams don't bother with the qualifier "UX" at all. Frequent, direct interaction with customers is just a fact of life for everyone on some teams, including the designers. As such, they don't have to model human behavior or claim user empathy as some departmental concern, like when you ask for a new VoIP headset from IT and have to put in a ticket or the IT guys ignore you for months. They're all just product designers designing a product.

To me, that's both ends of the UX maturity continuum - you just don't hear the word "UX" even used.

  • +1 The short answer is that the intention is for this to be a self-measure UX teams themselves, but I guess if done in a more formal way then it could be used to compared between different organizations (the same thing with how you can compare NPI across similar industries/sectors). – Michael Lai May 2 '18 at 5:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.