How to quantitatively compare two interfaces?

Question

I am in the process of building a tool to make some tasks easier than an existing application allows. Right now I am surveying users and receiving feedback but I want hard numbers.

The only quantitative measurement I know is number-of-actions to perform the same task. Are there any other measurements I can use?

Answer 1

There are many metrics you could use:

• Completion time.
• Divergence from ideal path for task completion, or how many steps does it take users on average to complete a task versus the optimal method.
• Completion time of novice users vs. trained users (you would need two tests for each interface here). This would test the intuitiveness of your interface.
• Quality of result - Does the task actually accomplish what the user wants to do?
• Using eye-tracking, you could track the number of saccades (eye-movements, or the distance the eye travels) per task.
• Demographic testing. If your target audience is broad, you should test the interfaces against the full spectrum of demographics and make adjustments accordingly.
• Heat maps for eye-tracking, mouse movement, or clicks. This allows you to identify points of interest/attention.
• Render/load speed of the interface.

There are also qualitative tests:

• Post-test survey. This would be another candidate for testing trained vs. untrained users.
• 5-second test. Show each interface for five seconds and ask the user to write down what they remember about it. This can help you make sure you're drawing their attention to the proper areas.
• Record the users during their tasks and have them talk through their thought process as they go along. Get their expectations of what they think should happen when they perform an action and compare it to what actually does happen.

Other tests could include:

• Color testing the design. Reduce it to grayscale and see if you're able to differentiate the different parts of the interface.
• Accessibility testing. How does it work with a screen reader?
• Depending on the application, you might also want to see how it performs on different screen sizes as well as touch interfaces.
• How does it work in different locales/languages? Does the interface accommodate changing the language?

Answer 2

I think Virtuosi Media has covered all the obvious metrics - the one obvious one that I'd add is the general completion rate ;-)

However since you say:

I am in the process of building a tool to make some tasks easier than an existing application allows. Right now I am surveying users and receiving feedback but I want hard numbers.

I find it vastly easier to go into a design process with an idea of what kind of thing I'm trying to improve. That way I can have a metric to guide me when it gets to validating the new design. You're trying to improve something - what? And how would you track it?