How to collect users' task completion times?

Question

I have to do a cost benefit analysis for a project in work. To get funding, I need to prove that the task to be fixed is time-consuming for users. We will only need to test about 20 users, but we'll need multiple responses from each user.

Should I just send something to all users where, for example, they click "Start" then "End" and it records the times or duration between clicks?

What other tools, strategies, or methods are available for timing users in this way?

Answer 1

I'm not sure what kind of application you're trying to test (web, mobile, desktop application, etc.), but it seems that you'll have to choose one of the 3 following options:

• Time the users yourself
  • Do public field-testing (also known as "guerrilla testing")
    • e.g. "Hello stranger, would you like a $5 gift card to {local coffee shop}? Our quick user testing will take less than 5 minutes of your time..."
  • Bring users on-site to test under observation
  • PROS: Direct observation gives you the chance to ask for additional feedback or elaboration on certain difficulties you may have observed, as well as opens the dialogue for the user to express any other concerns.
  • CONS: Requires face-to-face interaction, so your results may have a location bias. In other words, your results may not be representative of your broader target audience.
• Modify your application so it automatically collects these analytics for you
  • See Google Analytics for one example
  • PROS: Easy for you; works automatically (assuming you can integrate analytics into your application); usually provides much more insight into many different facets of how users use your applications, which can provide even more benefit.
  • CONS: User behavior not necessarily representative of a purely goal-oriented or task-driven approach to navigating the app, meaning your data may show more casual browsing behavior or distracted users than if you sat them down and observed them directly.
• Allow users to time themselves, using their own stopwatch
  • This option is perhaps the least reliable, but it's easy to collect responses from a wide number of people at once
  • PROS: Users can time themselves remotely, so your responses are not limited to users with whom you can physically interact.
  • CONS: Your data has to be taken with a grain of salt, as you can only trust that your users have completed the requested tasks, and have timed themselves accurately.
• As suggested by @ralien, you could utilize KLM (or "Keystroke-level modeling") (demonstration)
  • Allows you to estimate the time to complete the task without involving users at all.
  • PROS: Cheap, and provides results which may be good enough to satisfy your requirements.
  • CONS: Offers a theoretical analysis of an application's interface and organization, so may not align completely with users' actual experience.

In any situation, you'll likely have to incentivize your users in some way—most people don't like to spend their free time doing user testing just for the fun of it.

Answer 2

the question is bit unclear to me. My understanding is you want to do measure task completion for sample size of 20, and lest assume you have 10 tasks fro participants: There are two main way in usability testing: Moderated Unmoderated

For moderated, you need first recruit participants based on your persona. You could do this online, or face-to-face. To do this online you need some tools like Validately, user zoom, … or even you can use Skype or google hang out. The reason I mentioned special tools like user zoom, in this tools you can define tasks, and the tool measure time on task automatically. However to do this in cost effective way you can do this with other free software, and ask some one else from your team to join the session( notetaker), and with stop watch record the Time on tasks.

For moderated, Face-to face, you could ask participants to come to your office, and do the test, and measure the task. Probably you don’t have room with two way mirror, but still you could book a meeting room and again ask your colleague to take note, an recored the TOT. Note, first explain role of note taker for participants, as they would feel overwhelmed to see two person in the room, second only except you and note taker, don’t any one else be in the room, you could record the session, or if you have stockholder to be involve they should watch from live recording, or pre record later.

For unmoderated, there some website like usertetsing.com, that you could upload your prototype. This website has been designed for this, you upload the prototype, and then design the tasks. They will show this to the number of people that you would ask and they would come back with the results. You could do this with user zoom too.

Last but not least, I don’t recommend Guerrilla testing at all for time on time on task. There are may factors that could effect users effectiveness, on finishing a task. Unless you are doing a test that related to that context, example. You want to design something that people use in airport and you go to airport and see how all those factors in public effect finishing a task ( mix of contextual enquiry + ToT). But if you are designing a website, app,… that people use 90% in the office, home,… you should do this task under control environment.