5

I was setting up a usability test at my office where we have never done much formal usability testing. One of management's first questions was, "How long will this take?" I didn't have a good answer because I had been too busy thinking about writing scenarios, creating prototypes, recruiting participants, etc... to ever consider this.

Nielsen says a beginner could run a test in ~ 40 hours and I read an S&TC article (pg 7) saying that a good estimation formula was 10 hrs * # of testers * length of session. Both of these estimates seem reasonable but kind of out of date (Nielsen 1998, S&TC 2003).

Has anyone used these models before? Were they semi-accurate? Do you use a model you're quite fond of?

Note: I plan to track time spent on each task of preparing this usability test so I can use them as a base when planning future studies.

5
  • What are you testing? Are you testing a brand new application> A few newly create site pages? Or added functionality of some kind?
    – JeffH
    May 16, 2012 at 13:32
  • # of testers * Session Length is obviously valid but the 10 hrs is probably more accurately split out into your standard prep/analysis time per trial, which can vary a lot depending on the shop and the test.
    – Ben Brocka
    May 16, 2012 at 13:35
  • @JeffH There are some new regulations requiring us to test some existing functionality. I expect that we will save some time in being able to use our existing software instead of a prototype/mockup. May 16, 2012 at 13:45
  • @BenBrocka Splitting the 10 hours is definitely the challenge. I feel pretty comfortable with the process but there's a lot of bureaucratic weight to get moving as this is their first time with a serious usability test. Perhaps a high estimate and a healthy sized grain of salt is the best option here. May 16, 2012 at 13:49
  • 1
    Yeah, while it might sound bad on the face of it for your first usability test it's better that they're surprised it went so fast compared to your estimate as opposed to being angry it took longer.
    – Ben Brocka
    May 16, 2012 at 13:57

3 Answers 3

3

In my experience, the effort invested in testing will vary considerably based on the complexity of the software you're testing, whether your test materials are prepared already, the difficulty of recruiting participants and the number of personas you feel you need to text.

Complexity of Software - I tested an Internet Bill Pay / Banking app in 1996 when that sort of transaction was complex for many users. Consequently, they moved slowly through the app and it was easy to consume an entire session simply testing view account balance and transfer funds. More sessions were needed to test add payee, confirm payment was successful, dispute payment, etc

Effort Required to Produce Test Materials - Are you testing an existing app, paper prototypes, or an interactive prototype? The more you need to produce, obviously, the bigger the effort. In my experience, this is the largest variable that affects project duration.

Difficulty of Recruiting Participants - I'm currently testing an app (more LeanUX than usability but still relevant to this question) that has ~200 possible users nationwide. Each controls around $2 billion in spending annually. Getting their time has proven to be very difficult. Can anyone off the street be a test subject, or do you need to invest significant time recruiting test subjects with unique qualities?

Number of Personas - If you're testing 6 distinct personas, obviously this is much more time consuming than 2. It tends have a multiplying effect on the other aspects of testing - more personas often requires more screens in the prototype, more effort in recruiting and is an indicator of software complexity.

My Advice I can't tell you how to estimate the effort. But I can suggest an approach that's worked for me in the past. Borrow a trick from Agile software development and timebox. Tell management that you want to test 1 persona using simple paper prototypes of one particular feature. Test five participants and promise you'll get it done in 40 hours (this assumes that recruiting participants is relatively easy). Odds are you'll get the task done pretty close to the limit given to you. This allows you to return to management with a) some interesting results on how the app can be improved and b) a much better handle on what it takes to do usability testing with your team and your users on your app. This tends to make subsequent decisions easier and increases comfort levels with approving larger testing efforts.

1

For the most part it appears that the numbers you are quoting are still pretty valid. The biggest variable is going to be based on what you are testing. If you are testing existing functionality you will save a lot of time because you wont need to develop prototypes or mock-ups. For new functionality your numbers will be much closer to the ones you quoted.

If you are looking into decreasing your numbers you might want to look for a user testing service like Verifyapp. My organization has recently started using it for smaller user testing scenarios and it seems to be working out. The prices are reasonable and tests are very easy to setup.

Mind you this might not be the best software for your user testing. There is quiet a few options out there if you are looking for some automated solutions. These solutions normally speed up your setup and analysis phases.

1
  • Verifyapp looks very slick. I'll definitely check it out. May 18, 2012 at 13:22
0

You might be interested in this article by Jeff Sauro explaining the KLM model and how it can be used to provide an estimate for task completion time. The model is based on several key assumptions about user behaviour and competency (which is also based on previous research), but it does provide you with a basis to construct your own model and tweak some of the parameters. It can also be adopted for other interface types and interactions if you understand the fundamental principles behind it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.