I am working on calculating a "quality of experience" metric for an interactive application. This takes into account things like how responsive the app is, and how good the output is (sorry this is phrased somewhat abstractly, but the question is really not about the specific application). It is very easy to calculate how responsive the app or how good the output is right now (for example, in the past second).

The question: how to aggregate those instantaneous numbers into an overall quality of experience metric for a user session?

An average does not produce a very intuitive result. To see this, let us suppose the user used the application for an hour, and there were 10 individual problems (glitches, instances of bad user experience) each lasting a second. The average would be the same no matter how they were ordered in time (also very high: 99.7% of seconds are problem-free). But the experience may be quite different, if the problems

  1. occurred back-to-back in the middle of the one hour (10 seconds total), or

  2. were spread out every six minutes during the whole hour, or

  3. were spread out every minute for ten minutes

Can you please point to any research on how users rate intermittent problems depending on how they are spaced out?

  • 2
    One thing that might help is to look at these metrics from the context of individual tasks that the user is trying to complete. Remember that metrics are just there to help you get an overall idea, it doesn't replace understanding the interactions as they take place from a user's point of view.
    – Michael Lai
    Nov 2, 2013 at 5:03
  • 1
    This is a great question. But finding the answer needs research enough for at least one Ph.D., and as far as I am aware, nobody has ever done that. I have read some of the most recent publications which take baby steps towards such metrics, and nobody uses an established method for it, so while I haven't read all the literature on the topic, it is probable that such a thing doesn't yet exist.
    – Rumi P.
    Nov 4, 2013 at 11:16

1 Answer 1


This is an interesting question: really what you're talking about is that badly defined thing we call 'stress'.

Too much bad happening too quickly (situation 1 on the list) and people get stressed - and when people get stressed they start making bad decisions.

You can argue that 'stress' is a state of overarousal - and then this is covered by something called the Yerkes Dodson Law

As the graph shows, a bit of arousal / stress improves performance - but for complex tasks performance decreases once arousal / stress goes too high. If you watch people in this kind of situation they actually stop dealing with the information they are getting from their environment.

And of course, it is common sense that if you throw too much at somebody too quickly they'll just get mad with it and give up.

So in terms of an overall metric I'd add a (heavy)loading factor on any issues that happen back-to-back.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.