How to pick the right rating system?

Question

We are currently writing a web application which is used by one team inside the organisation. With this application they want to track the quality of the work done by others. (Quality assurance)

The problem is now that the client doesn't know exactly how they want to rate it. One idea is to use a - / 0 / + or another one is 1 / 2 / 3 / 4 / 5. (Or from 1 to 10.) Other ideas came up within the team as well.

We would like to help the client in finding the right rating system but we don't have much knowledge on how to choose the best system. Are there some general rules on when to use which system?

Answer 1

Unipolar vs bipolar

Unipolar scales

A system from 1..5 is unipolar. This is the system used by Amazon for reviews.

While 5 denotes 'excellent' and 1 'really poor', 3 is ambiguous - it doesn't necessarily denotes neutral. Some people will like the product, but will deduce 2 starts due to 2 cons.

Unipolar scales tend to ask How good? (or any other adjective), but don't provide a clear cut neutral.

This type of scale is appropriate when, say, marking someone's work - the marker can't really vote 'I'm neutral on the quality of this work', they must choose a magnitude.

Bipolar scales

A system running from -2..2, on the other hand, clearly signifies the bad range, the good range, and the neutral mark.

This type of scale asks How good or bad?, with an option to be neutral.

It is worth noting that there is potentially a bit more of a negative reinforcement in a mark like -2 (on a scale from -2..2) compared to getting a 1 (on a scale from 1..5). Ditto for an average mark of 0 (-2..2) compared to 3( 1..5).

That middle choice

Odd amount of choices, particularly on bipolar scales, allow a neutral choice.

There is an argument that an indecision is the outcome of the conscious brain taking shortcuts to reduce load, whereas our emotional subconscious must make a decision. In other words, regardless of what we say, we must have a preferences leaning towards one of the options.

What's more, a neutral view is not always informative business wise.

Amount of choices

A binary choice (yes or no) provide no place for degree. This is appropriate in some cases, such as when asking "Shall we go out tonight?"

Scales that allow degree are good for things that involve degree - "How clean was the toilets today?"

What is worth remembering is that the more options:

• The more people have to think about their choice, meaning increased cognitive load.
• The more blurry what each choice stand for - 7 could be seen as good for one but quite bad for another.

British undergraduate marking

Perhaps a good example for this is the marking scheme in many UK universities. While works may be marked on a 0-100% scale, they will also be awarded a mark between 1..16, which is subdivided into 4 bands:

• First (1..4, also 90%-100%)
• Upper second (5..8, also 80%-89%)
• Lower second (9..12, also 70%-79%)
• Third (13..16, also 60%-69%)

This seems overly complicated, but it is actually quite easy to place a work within a band first, then fine tune using the 1..4, 5..8, etc.

Having to put a mark as percentage is problematic since the scale is massive and there isn't real notion of how to break it down.

You'll find that when more than one person marks a work, they hardly ever choose a different band, whereas with percentages it is common to have marks more than 10% apart.

Summary

I guess that the easiest recommendation would be not to opt for long scales unless you make it very explicit, agreed and understood by everyone what each option mean.

I'd simply go for a unipolar 4 star system.