3

I have a usability test coming up and I want to test to see if one user journey performs better than the other.

It's for a task that you could do daily, ie; setting up calendar/reminder.

Essentially it's about power users vs casual users. One user journey is clearly simpler and would probably perform better doing the task once, but is a few more clicks.

However, you know how a task can become slightly cumbersome, and those additional clicks starts to annoy you a few months down the track. How do you test for something like that? It's not like I can ask this user to perform this task replicating 3 months worth of use in a 1hr session. I can ask the user to perform this task multiple times, but is that enough?

Thanks.

2

Looks like you need a diary study. Participants check in at pre-determined times with whatever info you need. They're, by necessity, unmoderated, but if you have participants who are motivated, you can get good results.

I've never performed one, but I was a participant in one a long time ago. I took photos (with a disposable camera!) and kept a paper journal every day, taking note of the activities they asked me to.

This was a preliminary study to understand how people performed a task so the design team would know what to design and what it should do. But doing a usability test this way should work too.

2

Excellent question - something that I've been asked quite a lot.

There's a couple of things you can do test this kind of experience.

  1. A comparison usability test. You'd create 2 tests, with approximately 5 testers in each. You'll have test 1, testing user journey A vs user journey B. Then, conduct test 2, testing user journey B vs journey A. You run tests this way to try and alleviate bias for which journey was easier/harder to use.
  2. A/B - run an A/B test with a 50/50 split of traffic to each user journey. You can run the test over a long period of time to see how many user's complete a certain journey or complete a certain action you're wanting to test.
  3. Setup session recordings. A session recording tool like FullStory will be able to record the screen of the user, without the participant knowing. You'll be able to see from the screen recording sessions whether a user is having trouble completing a certain journey, action etc.
  4. Setup tracking and record your retention. Depending on what you use as an analytics tool, you could set up event tracking for a particular feature and track its usage on a per-user basis and see if usage declines over a certain period of time. If usage is slipping, fire a poll to ask what they think of the feature. You could even ask a survey to ask for questions and dive deeper into why usage is dropping.

I hope this helps! If you have any questions, don't hesitate to comment back :)

  • 1
    Except for point #4, your suggestions don't really answer the core question which is how to test long term use. A/B tests or session recordings are short-term and won't answer the question of which UI is best after hundreds of uses. – celinelenoble Sep 7 '18 at 18:27
  • Thank you. Yes point 4 has some helpful advice for usability testing once a feature is released to users. – Rhys Sep 13 '18 at 5:23
  • I'm going to disagree with you on the point about session recordings being of short-term use. If you're able to see how users are using your product in a certain period, say month 1, and then analyse sessions recordings fortnightly on how they're using your product, you'll see whether or not users are still interacting and behaving the same way after month 1 compared to say month 3. – Preston Sep 14 '18 at 7:52
1

Great question because power users are often forgotten about when considering usability testing.

You have two different types of users.

These two types are commonly called 'novices' and 'experts'.

Novices

A novice is anyone who is seeing your application for the first time (or one of the first times). This group is still deciphering what your interface is communicating regarding what is possible (the features and content available to them), your application's structure (the information architecture) and how to navigate through it and interact with it all (the interface). They could be subject matter experts, but they are novice in the sense of not having had exposure to your application.

When you test this user, you are typically testing 'learnability' - how well they do the above mentioned and complete tasks as new users of your application.

Experts

An expert has interacted with (or seen documentation or had training for) your application before, enough that they understand the steps needed to complete the tasks you're interested in. This is as you put it, your 'power user'.

When you test this user, you are typically testing 'efficiency' - how quickly and effortlessly they can complete a task given they already know how to do it.

What you're testing is different (and thus your approach has to be).

I think you probably have the novice user testing for learnability under control. This is the type of testing most UX Designers / other practitioners are familiar with - getting newly recruited test participants to work through tasks, talking out loud as they go for further insights.

Expert users and testing efficiency

Without users (predictive)

  • KLM/TLM GOMS - This is the process of counting the number of interactions a user would need to complete a task. Each type of interaction (like a movement of the mouse, a swipe of the thumb, a pressing of a key) is given a time cost, and then summed together to give an approximate total time to complete a task. This assumes the task is completed with no issues encountered and no detours taken (which is why this method only makes sense in the context of expert users). You would use this to compare your two designs and see just how much of a difference there really is.

With users

  • User testing (but done seemingly incorrectly) - Where we've all learnt not to help the user, that goes out the window. Make your test participant an expert by providing them training resources (like how-to guides), prior access to the designs/prototype/application, and conduct a walkthrough where they can ask further questions. Also, rather than your typical think aloud session, don't instruct the participant to talk out aloud. While Jakob Nielson disagrees with that (advocating for still getting them to do so) he also says "Skilled behavior is often automated behavior... when people are unaware of how they think about a certain behavior, they can't verbalize the reasoning behind their actions." My preference is to only prompt them if i think the pause or hesitation warrants explaining, otherwise ask afterwards.
  • 1
    Rather counter-intuitively, in some situations 'novices' will perform better than 'experts': if you give 'experts' a car to drive with the foot pedals in a different order than normal, they have to unlearn their previous expertise. Novices don't have that problem. – PhillipW Sep 9 '18 at 13:32
  • @PhillipW - Yep makes sense definitely. I'm using 'expert' relative to the interface being tested though. So in that example, no one is an expert because that particular setup is new to all of them. – Zenon Sep 9 '18 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.