2

I'm making a user study. Users had to perform the same task 10 times. I've recorded the completion times of each task. How could I calculate the UI's efficiency based on this data? Does anyone know the correct formula to do that? Preferably with references. Thanks!

UPDATE:

I need to calculate the efficiency time. The user always starts from 0 efficiency and then through time (number of tries) efficiency increases and efficiency time decreases both logarithmically reaching its final value.

I've calculated the perfect efficiency time using the Keystroke Level Modeling and it's, let's say 6s. Which means in perfect case scenario a "robot" user will complete this task in 6s. But since I'm working with people the efficiency time differs from user to user and I need to acquire those efficiency time values to average them and compare with the robot user efficiency time.

So I have the data for 10 tries of the same task and I need to get this efficiency time value for each user to sum up the experiments. I reckon there should be a statistical tool to calculate this, and this is what I'm looking for.

UPDATE 2:

I've found this link with a statistical approach to get efficiency over time, overall relative efficiency, expert relative efficiency. The problem it has absolutely no references whatsoever so I'm not sure I can use that as it is. Did some of you maybe faced this approach before and could introduce some references.

Please ask if you need more clarification.

  • 2
    How are you defining efficiency? Speed of performing each task? The number of tasks users can perform in a set amount of time? The number of clicks to complete a task? The ability of users to perform multiple tasks in parallel? (It's also useful to consider whether or not efficiency is a useful metric.) – Ken Mohnkern Oct 21 '15 at 11:37
  • 1
    Is this a frequently-performed task? Is it normal for a user to do it many times? – user31143 Oct 21 '15 at 11:47
  • @KenMohnkern Thanks for your comments. I should have mentioned that in the question. I need to calculate the efficiency time. As you can imagine the user always starts from 0 efficiency and then through time (number of tries) efficiency increases and efficiency time decreases logarithmically reaching its final value. So I have the data for 10 tries of the same task and I need to get this value to sum up the experiment. I reckon there should be a statistical tool to calculate this value, this is what I'm looking for in fact. – LoomyBear Oct 21 '15 at 12:10
  • @dan1111 It's a task designed solely for research purposes. You know this kind of academic tasks that makes no practical sense. And no in real life user will never face this task. So I guess it's not relevant. – LoomyBear Oct 21 '15 at 12:12
  • 1
    Godspeed, @LoomyBear. – Ken Mohnkern Oct 21 '15 at 13:15
3

The question for me is here: Can you define key metrics for success?

I do not think that there is a formula (at least I know none) that tells me "The interface works great!". Usually, the question is about segments, cohorts & success metrics.

For example: Assuming your page is a product page and you want to measure how often in the 10 visits users added something to the wishlist. Then measuring success for me would be "how does the add-to-wishlist micro conversion rate evolve during the 10 visits?". Accordingly to user behavior, I would compete the control version with another one and see the difference to the original version. With this, I can see "Version B works 5% better than A", which makes "the interface perform better than the previous version" - but that is a more regular usability testing approach.

Second possibility is: Search for studies on this type of page's performance - compare it to industry average. This might be expensive though.

So 1. Option: Comparison to my own previous version to see if it is performing better or worse. 2. Option is comparison to industry standard.

  • Thanks for your answer. I should have mentioned that in the question. I need to calculate the efficiency time. As you can imagine the user always starts from 0 efficiency and then through time (number of tries) efficiency increases and efficiency time decreases logarithmically reaching its final value. So I have the data for 10 tries of the same task and I need to get this value to sum up the experiment. I reckon there should be a statistical tool to calculate this value, this is what I'm looking for in fact. – LoomyBear Oct 21 '15 at 12:09
1

I don't really think this sort of metric in the way you've outlined exists. The fact that you indicate efficiency for the first attempt ought to be 0 is a bit bizarre. It's implies the user is not at all efficient, when in fact, the task could have been so easy, that the user was flying through it even in the 1st attempt.

It'll make more sense to look at performance improvement over time.

E.g. Plot Task Duration over Time (# of trials)

mockup

download bmml source – Wireframes created with Balsamiq Mockups

You should be able to get a regression line to fit the graph.

Do this for all your test condition groups and analyze differences in the regression fit lines.

  • @nightting You're right 0 efficiency assumes that the task will never be done. Sorry for that. What I meant that the user is far from being really efficient on the first attempt. She needs time to get to the optimal/working efficiency. As you referenced in the image the time of completion always gets exponential over time and the part of the graph where it gets flatten hides this efficiency time which I want to obtain. – LoomyBear Oct 22 '15 at 0:36
  • I have all this graphs built already, but does just the graph (visual) comparison gonna be enough for a valid scientific conclusion? I need a statistical value, which I call the "efficiency time" that will allow me to compare (statistically) two stimuli and conclude which one performed better. – LoomyBear Oct 22 '15 at 0:40
  • @LoomyBear That's what I thought. Okay, I'm out of my depth with the statistical analysis side if your regression line fit is non-linear, which I suspect will be your case here. This CAN be done though. Your best bet would be to take your dataset to the math department and ask them: I have generated regression lines based on two set of data. I want to perform statistical analysis to see if there's any statistical difference between the two. How should I test this? Show them your graphs. I suspect the tests used will be different depending on the type of fit. – nightning Oct 22 '15 at 4:46
  • Also tell them if multiple data points come from the same tester. You might want to try asking this on the Math SE. – nightning Oct 22 '15 at 4:47
0

What you may be looking for is called GOMS Modeling. Goals, Operators, Methods and Selection rules https://en.wikipedia.org/wiki/GOMS

Loosely, it uses things like Fitts' law (the time it takes to acquire a target is a function of size of the target and distance to the target)... and Hick's Law (the time it takes to make a choice is a function of the number of choices) to estimate the time of your UI.

You may have to adapt it to your case. When they (Card, Moran and Newall) wrote their paper, they were using a mouse as the input device. If you are on mobile, you may need to run some tests to come up with your own constants for some of the variables.

  • I know about GOMS and FItt's law (as you noticed I referenced the Keystroke Level Modelling (KLM) which is a older brother of GOMS). The thing is it's a prediction model and what I'm looking for is a value to sumarize the obtained real life data. I don't know why but I call it the efficiency time. Maybe I'm totally wrong here. Feel free to correct me. – LoomyBear Oct 22 '15 at 0:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.