Has anyone used the Lostness metric? I'm trying to use it to benchmark a navigation scheme. One of the variables (R) is the minimum number of screens that must be visited to complete a task. I calculated R starting at the homepage for each task. The software we use to run unmoderated testing begins each task (after the first) on the screen where the participant completed the previous task. In this case, is R still valid to calculate the metric? It seems like it wouldn't be because participants are all beginning tasks on different screens and not the homepage. I want to make sure I'm thinking about this the right way. Thanks!
Interesting question! I have never used
Lostness directly myself, but I think it is worth taking a step back and look at the bigger question:
What are we trying to achieve?
Metrics don't exist in a vacuum! When you use a metric, you are trying to use it as a shortcut to measure something else. This may sound very obvious, but I think you should first be really clear about for which question you are trying to find a "valid metric". Could be any of the following or something else:
- Are my users feeling significant confusion when using my app?
- Do I have unnecessarily complicated flows in my app?
- How many interactions with my app fail due to confusion?
...and so on. They might be similar questions, but they are not the same. Next, you also need to specify a goal of what are you trying to achieve, because that also plays into what a "valid" metric is. Is your goal for example...
- ...reaching a certain standard of ease-of-use
- ...have a better navigation than a competitor
- ...find out what the most problematic flow is to work on first
...or something else?
How to measure Lostness and why
By "Lostness" you probably mean this formula, right?
L = Lostness
N = The number of different screens visited during the task.
S = The total number of screens visited during the task.
R = The minimum number of screens that must be visited to complete a task.
Well, this defines
R as the actual minimum number of screens that must be visited, so in that sense your interpretation is not valid, because by your description is it not necessary to start at the homepage to complete a specific task, so you are adding unnecessary steps to the process.
Now the ironic part is if you actually hunt down the origin of this random formula every blog post is so happily citing (I am pretty sure it is Smith, P. A. (1996). Towards a practical measure of hypertext usability ), you notice two interesting things:
- The author considers a lot of different measures for the feeling of being lost, efficiency of task completion and what not (listing them would completely blow this already too long answer out of proportion). The formula you see everywhere for "Lostness" is basically randomly selected.
- The author specifically tries to describe "Lostness" in pure hypertext information systems (think Wikipedia) and not for systems where there is the assumption that, and I quote "a particular task being undertaken, that there is a ‘correct’ way in which this task should be performed and that the purpose of the system is to enable the user to carry out the required task as quickly as possible".
So in a way, Lostness is an invalid measurement for your project!
So, is the way you are measuring Lostness/R valid?
Yes and no.
- If you are just trying to rank your tasks from most-complicated to least-complicated, this is going to be totally alright. Go ahead.
- If you are trying to find things that are way too complicated, you will still find them because they'll stick out like a sore thumb even if one of your measures is a li'l muddy. Go ahead.
- If you are trying to achieve a certain standard of (minimal) Lostness, you need to take a measurement of
Lthat is reasonable not-lost, and compare all your screens against that. Do not use the 0.4 threshold that is randomly flowing around on Medium, that is yet another random paper result that is hardly relevant to your use case. Will be a tiny bit of statistical interference because again your
Ris a bit muddy but will be a reasonable way to get to an easily understandable app. Go ahead.
- If you have a very good reason to stick to Smith's Lostness to the letter, then no, your method is not valid. But I do not think that you are in this situation.
Sorry, long answer for a short question. I hope it aids you in your journey towards good usability measurements.
TLDR: Metrics ain't magic, even if the formula looks really cool. Always step back and ask yourself what you want to find out.
I haven't heard of this metric but this should not be different from any other testing metric or method: You want unbiased results that are comparable.
The user starts from a page and needs additional steps to get to the starting point for the next task. Do you add those extra steps to the minimum required steps?
If all participants are guaranteed to start on the same page, the answer is yes. Those extra steps are part of the minimum required steps but you need to redefine the actual starting point. You can compare results between participants knowing they had the same starting point.
But if participants can get lost in the previous task and it is uncertain from which page they will start the next task, you can't compare how well they perform on their next task with other participants that started from a different position. What happens when you do this is that you suddenly measure the entire set of tasks as if it were one task instead of all the tasks individually. If this is the desired result, you need to add up the minimum number of steps for all tasks and count all steps through all tasks to get the necessary variables.
If you want to use the metric on a per task basis, make sure that the test resets or that participants are taken to the right starting point before they start the task.
I also recommend to look at the big picture first before using any metric as mentioned in @koljapluemer's excellent answer.