Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I have been looking to develop some metrics for a UX project (which was also a question I posted previoiusly), and have been reading about the Kano model and how it maps to different user behaviour/expectations. I was wondering whether it is practical to dissect the functions/features of a product based on user perception about the presence and absence of features (ref here) and tallying up to see if there were more features that generated excited rather than expected or dissatisfied experiences.

There have been proposals of frameworks for evaluating user experience (e.g. Google's HEART framework) but these models do not provide actual metrics or mapping to the components or categories that make up the whole user experience which we want to capture. I have found that the Kano model is one starting point from mapping product functions and features to particular aspects of the user experience, and would like to look for other ways to start filling in the rest of the gaps.

The question is, can the features of a software product or service be classified as one of the three categories and weighted to provide a value/metric for how users feel about a product. And is it practical to track and monitor this over the lifecycle the product to provide one aspect of the user experience?

share|improve this question
    
Interesting question and I wish there were more answers. Maybe it motivate more people to answer if some more of the previous asked questions had accepted answers? –  greenforest Jul 30 '13 at 22:42
    
@greenforest - I didn't really get too many answers from my previous, but maybe looking at a bounty or the UX SE board for some inspirations –  Michael Lai Jul 31 '13 at 1:27
    
I have experience using the Kano model with UX, and particularly with using the survey method to gather actual metrics. There is definitely value in this. –  Erics Jul 31 '13 at 9:10
    
@MichaelLai I was more referring to the ratio of 33 questions asked and only 3 of them with accepted answers :) –  greenforest Jul 31 '13 at 9:27
    
I have been thinking about measuring user experience for some time. Haven't had chance to put my ideas into a legible form yet. Interested to see what people have to say –  Sheff Jul 31 '13 at 10:05

5 Answers 5

up vote 8 down vote accepted
+50

Short version:

  1. Yes, you can use Kano Analysis to arrive at a metric for how users feel about a product. Actually, two metrics, but possibly not exactly what you want.

  2. Yes, it is practical to re-test and monitor this metric over the life-cycle of the product. Actually, you should.


The long version... get a cuppa, settle in to a comfy chair...

Derive a metric from Kano Analysis?

The question is, can the features of a software product or service be classified as one of the three categories and weighted to provide a value/metric for how users feel about a product.

You could, however it would be a coarse metric, and it would more properly describe how users feel about the potential of the product.

And it really should be two metrics.

The Kano model is a variation on Herzberg's Two Factor theory of satisfaction, which states that there are certain factors that cause satisfaction while a separate set of factors cause dissatisfaction. That is, satisfaction and dissatisfaction act independently of each other.

To quote Jesse James Garrett: "Reduction of misery does not equate to increase of joy".

So, the two metrics you want to extract is the overall impact on satisfaction, and the overall impact on dissatisfaction.

If you were to try to distill this down to just one metric you should give at least a 3x weighting to all the features which appear to be baseline (aka basic, must-be, threshold, et al). See this TechCrunch article for references to research justifying the 3-5x weighting to give to negative things.

Also, not all baseline features have the same degree of impact, and also for different numbers of users. Kano analysis can help with the latter easily, but the former is only implied in the method.

You could ask an extra question on each feature to evaluate relative impact, or incorporate Choice-Based Conjont analysis. It's not as important as you might think, as nett importance would be determined by multiplying the relative importance by the proportion of users that feel that way. If you have 10% who think feature X is really, really, really nifty ... that's probably not as important as 80% who think feature Y is kinda, yeah, mostly nifty.

It's important to note though that the metrics from Kano Analysis only reflect potential for satisfaction and dissatisfaction. The model is built on the idea that the user's response to a feature varies according to the degree of execution of the feature, with the three main Kano classifications describing the shape of the response line.

So, the classifications and thus the metrics you get from Kano analysis won't tell you how good your product actually is (you need to use other survey methods to get that). However, it will tell you if there's more you can do, and also help you focus your research efforts appropriately.

Track and monitor over time?

And is it practical to track and monitor this over the life-cycle of the product to provide one aspect of the user experience?

Yes. You would gather the quantitative data for Kano modelling by surveying a random sample of your users, and you could do so periodically and thus monitor it that way (that's the beauty of statistical sampling). Just don't keep surveying the exact same sample each time.

If the length of the survey worries you then you can also survey different random samples of users, each with a short sub-set of features. The total number of survey participants would go up of course, but the effort requested of them individually is still low. This works with Kano Analysis since each feature is evaluated independently of others.

You could also refine the set of features you survey over time. If for example you find that 99% of respondents consider Feature X to be of no importance then simply stop asking about it. Also, if your team has come up with some genius idea for a new feature you can throw it in the mix and see it mapped against the other features. If it's a stinker, best to find out early and drop it before expending any further resources on it.

Importantly, you should monitor the Kano classifications of your features over time. Part of the Kano model discusses the fact that feature classifications drift over time, starting out as Delighters, then become Performance features, before settling down as Baseline features.

You want to know when a Delighter becomes a Performer because according to the model that feature will not only contribute to satisfaction but insufficient implementation will now contribute to dissatisfaction. Now, it's important to improve the feature, and not simply rely on mere inclusion to drive overall satisfaction. (Remember when we were all excited about our phones having a camera built in, despite how crappy they were? Phones in cameras are no longer a delighter, they need to be of quality.)

Similarly, when a feature fades from being a Performer and becomes a Baseline you want to reposition the feature - it's no longer something that should be trumpeted and it shouldn't sing and dance .. it should simply do what's required and then get out of the damn way. No fancy gift-wrapping.

You also want to look into whether particular large changes in the market landscape are impacting the importance of various features - a given feature might suddenly accelerate its drift from one classification to another, or even simply just jump right on over to the next classification. Look to geo-political shenanigans for example situations.


Let me wrap up by describing a couple of examples of how I've used Kano Analysis.

Case study 1:

We were building a very complex online application, and there was the usual overly-long list of feature requirements. We needed a way to get a handle on which features are absolute must-haves, and which we could delay for later.

So, we worked up a list of 41 features and then used a panel service to recruit 45 participants for a Kano survey, offering a sizable incentive as compensation.

We crunched the numbers, and this was the result:

users response to features

The horizontal axis is the proportion of participants that would negatively react to an insufficient implementation of each feature, while the vertical axis is the proportion of participants that would be positively affected by the feature.

  • The grey dots are features for which most respondents didn't consider important for either satisfaction or dissatisfaction. That list of features got dumped into the backlog real quick. Some were borderline, and we kept an eye on them for later.

  • The blue dots are the Baseline features. Again, we simplified the design to focus on just the core functionality of each. What would it take to ensure the users would not hate us?

  • The green dots are the performer features. Those we knew we'd need to pay further attention to. They would have to meet the basic expectations of the users, and from there any further refinement would provide a satisfaction payoff.

  • The yellow dots are features most respondents considered as delighters. Those features we simplified and scaled back the designs. We also cherry-picked the ones that would be drop dead simple to do and which were real differentiators, knowing that most users wouldn't even notice the other delighters were missing (by definition). The standard we had to reach was What would it take to get the attention of the users?

We didn't distill all that data down to a single metric as we were already getting a lot of value out of the map as is. We could have, if we wanted a very high-level strategic metric (similar to how NPS is a high-level metric).

Case study 2:

This was a smaller and shorter exercise. We had conducted a series of stakeholder workshops and user workshops, and from those we developed a list of ideas for features for a new website. As you know, small samples combined with group dynamics meant we simply couldn't rely on the enthusiasm of the workshop participants to define the importance of each feature, and so we ran a Kano Survey for validation.

This time we used ethn.io to do the recruiting for us. We installed the javascript onto the client site and in less than 3 days we had 73 qualified participants.

This is the graph of the analysis:

users response to features

See that grey dot down in the bottom left? That was a feature everyone in the workshop thought would be pretty nifty. Turns out computer says no.

The position of a few other features also proved to be food for thought.

share|improve this answer
2  
shameless plug: I'm building a tool to conduct Kano surveys and do all the heavy lifting in the analysis. Find it at kano.io -- it's still in pre-beta testing. –  Erics Aug 1 '13 at 11:19
    
very helpful answer, and it really helps to illustrate the application of theory into practice. What is your particular role and have you found the data strong enough to convince stakeholders? Have you found any shortcomings or pitfalls that require other data to help interpret or explain? –  Michael Lai Aug 1 '13 at 22:46
    
Usually the big challenge is selling the results to project stakeholders, especially having to explain the concept that satisfaction isn't a one-dimensional scalar but is instead two separate factors. Assuming that though, incorporating CBC to measure importance of each feature is helpful to some degree. –  Erics Aug 1 '13 at 23:33
    
How is the Kano survey testing tool coming along? Would be interested to see if you have any stats on it. –  Michael Lai Jul 16 at 23:18

Maybe not a direct answer to your question but thoughts that I hope can add some value. Anyway, I'd love to see some more answers to the question.

ISO 9241-11 "Usability"

"The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use."

Kano

Differentiating product features of a product in basic attributes, performance attributes and excitement attributes.

Potential correlation

  • To fulfil the basic need the product needs to be effective.
    e.g. a task manager app needs to be able to store my tasks

  • A performance need might be the desire to achieve the goal in an efficient manner.
    e.g. the bespoke task manager app should enable me to add a task from anywhere in the app with just 2 clicks

  • The excitement, even though maybe not a feature itself, can increase the satisfaction and serve as differentiator over similar products that otherwise are also effective and efficient.
    e.g. the task manager has a beautiful designed UI and is hence a pleasure to use

share|improve this answer
    
Most of the things I have read about UX metrics seem to relate back directly to usability metrics. I do agree that there seems to be a good correlation, so it is interesting that I haven't seen the Kano model mentioned much in the HEART framework or other types of UX testing/evaluation discussions. –  Michael Lai Aug 1 '13 at 7:37

ISO/IEC TR 9126-3:2003 can be used for defining internal metrics - an internal evaluation. Seems the best option for your use. It is used for predicting the extent to which the UI can be understood, learnt, operated, attractive and compliant with ISO usability regulations and guidelines. For example, it breaks the UI down into question such as:

  • What proportion of the product functions are evident to the user?
  • What proportion of interface elements are self-explanatory?
  • What proportion of operations behave the same way to similar operations in other parts of the system?

A formula is used to generate an overall score. You can use Excel, and it doesn't take a UI expert to run this test.

share|improve this answer
    
But this is still looking at user experience from the perspective of the interface design. I think the Kano model is more relevant in terms of capturing some of the semi-qualitative aspects of how a user feels (i.e. what they experience) using the application. –  Michael Lai Aug 1 '13 at 7:30

Kano Model is representation of metrics to assess user satisfaction with a product. Satisfaction is very important measure but user experience is one step ahead of it. The historical evolution is from service quality --> Customer Care and Satisfaction --> Customer Experience. UX/CX is holistic concept which takes usability measures, aesthetic, emotions, cognition and experiences. In short, Kano Model is good as long as your focus is to gauge user satisfaction. But for user experience, you need more factors, otherwise your model will be confounded.

share|improve this answer
    
Point taken. This is why I asked whether it should be used to provide an indicator for one aspect of the user experience of the product/service. –  Michael Lai Jul 31 '13 at 22:53
    
Also, I think the Kano model delves a little bit deeper than simply satisfaction. It is semi-quantitative because some of the satisfaction/expectation scale to specific aspects of the product function or feature. –  Michael Lai Aug 1 '13 at 7:34

"And is it practical to track and monitor this over the lifecycle the product to provide one aspect of the user experience?"

No, because you certainly have many features and you need 2 questions for each to get the Kano state of it. That will be a terribly long survey, only to get the satus quo of your feature satisfaction. UX is more than product value and satisfaction, so I recommend to invest your energy into a concept of metrics, which gives you insights where to step further.

share|improve this answer
    
If we are simply looking at the cumulative or aggregate values then you can probably ask half of the users the questions for the feature when it is present and the other half for when the feature is absent. –  Michael Lai Aug 1 '13 at 7:32
    
Michael, you can't split the effort that way as the Kano Model compares the two answers to arrive at the classification. What you can do however is ask each participant about a random subset of all the features, thus reducing the per participant effort. –  Erics Aug 1 '13 at 13:06
    
@Erics, I agree it doesn't quite make sense to do that. But doing it that way means that you don't get a sense of the relative weighting that they user assigns to the features, which I think would be something of interest if you are collecting this type of information. –  Michael Lai Aug 8 '13 at 6:33
    
For an individual user you could ask them to rate the importance of the feature on a scale (in addition to the two kano questions). Don't bother asking if the two kano question result in an indifference result). –  Erics Aug 8 '13 at 8:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.