Defining "Instantaneous" as part of usability acceptance criteria

Question

The Setup

I have run into this problem time and time again. An interface is performing slow and QA and/or Engineering come back and ask, "can you please define the acceptance criteria for how fast it needs to perform."

Often the answer is (or should be) Instantaneously, however this is often met with, "well we need a number." or similar comment. I agree instantly is up to interpretation, may be unhelpful, etc. However, me picking a number out of thin air like 50 milliseconds or 250 milliseconds is just as unhelpful as QA will have no way to validate this.

It usually comes down to the old adage "I'll know it's right (or wrong) when I see it"

So my question to you is:

The Question

How do you define "instant response" as part of your usability acceptance criteria when speaking about performance. What has worked, and what has not worked. If you did set a exact number, how was it tested or confirmed? What is instant reaction time?

An example situation

You have a grid that is editable and need to define how long it should take when you click to edit (how long till the cell displays as an input) and how long it should take to commit the change when exit the field (how long it takes to turn into or appear as plain text).

Answer 1

The general rule of thumb for usability is to start off with no feedback, but to then display some busy indicator after 200ms, and if the process normally takes 5 seconds or more to present a larger feedback element (usually with a time elapsed timer, but preferably not with a progress bar unless you're very sure how long it will take). If something is likely to take more than 30-60 seconds to complete then you should consider adapting the architecture to support queuing/background processing so the user isn't forced to watch and wait.

If the feedback is meant to be part of an interactive process, like typing on a keyboard and then seeing the keys appear, then you really should be striving to reduce latency/feedback to under 100ms

There's another marker too - it's been found that users make a sub-conscious evaluation of a product in the first 50ms of what they see.

There's another angle to consider too - some accessibility assistive technologies won't see changes to a page if they occur after a delay .. the way they are built is they capture and cache the contents of the page once it is loaded, and then use that cached copy to present to the user. [Specifically JAWS, I can't find the link right now]

Answer 2

This is a very interesting question. I think that usability requirements like "instantaneous" by itself is pretty much worthless. You should have verifiable criteria for responsiveness. However you should also beforehand make sure that whatever verifiable criteria you set should be possible to measure and achieve. I'd imagine that a lot of response time measurements could be scripted. I'd also consider measuring standard deviation as consistency in response times is important for good user experience.

In general I agree with Erics that for instantaneous response times you should aim to under 100ms. A good reference for response times is Jakob Nielsen's Response Times: The 3 Important Limits.

Anyways the main point is that if you want that (usability) requirements are realized, you will need to set verifiable targets. I recommend checking the relevant literature. Couple articles related to the subject are:

• Methods for quantitative usability requirements: a case study on the development of the user interface of a mobile phone
• To what extent usability truly matters?: a study on usability requirements in call-for-tenders of software systems issued by public authorities

Answer 3

The Nielsen Norman Group cites usability guidelines that have been around since the 70s:

0.1 second is about the limit for having the user feel that the system is reacting instantaneously, meaning that no special feedback is necessary except to display the result.

1.0 second is about the limit for the user's flow of thought to stay uninterrupted, even though the user will notice the delay. Normally, no special feedback is necessary during delays of more than 0.1 but less than 1.0 second, but the user does lose the feeling of operating directly on the data.

10 seconds is about the limit for keeping the user's attention focused on the dialogue. For longer delays, users will want to perform other tasks while waiting for the computer to finish, so they should be given feedback indicating when the computer expects to be done. Feedback during the delay is especially important if the response time is likely to be highly variable, since users will then not know what to expect.