How to calculate user interface's efficiency time

Question

I'm making a user study. Users had to perform the same task 10 times. I've recorded the completion times of each task. How could I calculate the UI's efficiency based on this data? Does anyone know the correct formula to do that? Preferably with references. Thanks!

UPDATE:

I need to calculate the efficiency time. The user always starts from 0 efficiency and then through time (number of tries) efficiency increases and efficiency time decreases both logarithmically reaching its final value.

I've calculated the perfect efficiency time using the Keystroke Level Modeling and it's, let's say 6s. Which means in perfect case scenario a "robot" user will complete this task in 6s. But since I'm working with people the efficiency time differs from user to user and I need to acquire those efficiency time values to average them and compare with the robot user efficiency time.

So I have the data for 10 tries of the same task and I need to get this efficiency time value for each user to sum up the experiments. I reckon there should be a statistical tool to calculate this, and this is what I'm looking for.

UPDATE 2:

I've found this link with a statistical approach to get efficiency over time, overall relative efficiency, expert relative efficiency. The problem it has absolutely no references whatsoever so I'm not sure I can use that as it is. Did some of you maybe faced this approach before and could introduce some references.

Please ask if you need more clarification.

Answer 1

The question for me is here: Can you define key metrics for success?

I do not think that there is a formula (at least I know none) that tells me "The interface works great!". Usually, the question is about segments, cohorts & success metrics.

For example: Assuming your page is a product page and you want to measure how often in the 10 visits users added something to the wishlist. Then measuring success for me would be "how does the add-to-wishlist micro conversion rate evolve during the 10 visits?". Accordingly to user behavior, I would compete the control version with another one and see the difference to the original version. With this, I can see "Version B works 5% better than A", which makes "the interface perform better than the previous version" - but that is a more regular usability testing approach.

Second possibility is: Search for studies on this type of page's performance - compare it to industry average. This might be expensive though.

So 1. Option: Comparison to my own previous version to see if it is performing better or worse. 2. Option is comparison to industry standard.

Answer 2

I don't really think this sort of metric in the way you've outlined exists. The fact that you indicate efficiency for the first attempt ought to be 0 is a bit bizarre. It's implies the user is not at all efficient, when in fact, the task could have been so easy, that the user was flying through it even in the 1st attempt.

It'll make more sense to look at performance improvement over time.

E.g. Plot Task Duration over Time (# of trials)

^{download bmml source – Wireframes created with Balsamiq Mockups}

You should be able to get a regression line to fit the graph.

Do this for all your test condition groups and analyze differences in the regression fit lines.

Answer 3

What you may be looking for is called GOMS Modeling. Goals, Operators, Methods and Selection rules https://en.wikipedia.org/wiki/GOMS

Loosely, it uses things like Fitts' law (the time it takes to acquire a target is a function of size of the target and distance to the target)... and Hick's Law (the time it takes to make a choice is a function of the number of choices) to estimate the time of your UI.

You may have to adapt it to your case. When they (Card, Moran and Newall) wrote their paper, they were using a mouse as the input device. If you are on mobile, you may need to run some tests to come up with your own constants for some of the variables.