2

Under the current system I am using it is 2/1/-2 however I am beginning to doubt how fair this is; if something has 4 passes and 1 fail then surely that is 80% pass? Yet with this way of calculating points it would only be 60%.

Are fails so critical that they deserve to be punished in this way?

There is the consideration though that if a fail were to be worth 0 points then that would put a fail on a par with not relevant (e.g. a rule speaking about how to format tables on a site that has no tables). And it seems really unfair for a not relevant to be equal to a fail- on percentages this can be hidden (a 0 that counts towards the score vs. a 0 that is ignored) but in terms of brute points it will stand out.

As you can tell here I am really thinking a lot about a less important side of heuristic evaluations, the most important point is of course to find errors. but still, stake holders like numbers.

Are there any recommendations for scoring of heuristic evaluations?

1

Heuristic evaluations can be as simple or complex as you need it to be, and the general rule of thumb is to make it as simple as possible and to provide an actionable outcome when something fails.

In your case, if you are finding that a simplistic scoring system doesn't work for you, then you can try a number of things:

  1. Alter the scoring system from 2/1/-2 to something else?
  2. Allow for a greater range of values so that there are degrees of fails and passes
  3. Weigh the scoring against another criteria, like the impact (e.g. number of people it affects, the importance of that particular item relative to others)
  4. Weigh the scoring against the effort required to address the item
  5. If one item fails because of its link or dependency on another item, you can also take this into account to reduce the weighting of that item.

But as you can see, taking any number of those strategies into account can yield a more accurate score that reflects the state of the system (but also make the end result more difficult to digest and understand), but the real measure is probably against a more user-centric metric.

The main thing to remember about doing evaluations is that you shouldn't rely on one source/method to try and understand the whole picture.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.