What is a good method/formula for estimating how much time a usability test will require?

Question

I was setting up a usability test at my office where we have never done much formal usability testing. One of management's first questions was, "How long will this take?" I didn't have a good answer because I had been too busy thinking about writing scenarios, creating prototypes, recruiting participants, etc... to ever consider this.

Nielsen says a beginner could run a test in ~ 40 hours and I read an S&TC article (pg 7) saying that a good estimation formula was 10 hrs * # of testers * length of session. Both of these estimates seem reasonable but kind of out of date (Nielsen 1998, S&TC 2003).

Has anyone used these models before? Were they semi-accurate? Do you use a model you're quite fond of?

Note: I plan to track time spent on each task of preparing this usability test so I can use them as a base when planning future studies.

Answer 1

In my experience, the effort invested in testing will vary considerably based on the complexity of the software you're testing, whether your test materials are prepared already, the difficulty of recruiting participants and the number of personas you feel you need to text.

Complexity of Software - I tested an Internet Bill Pay / Banking app in 1996 when that sort of transaction was complex for many users. Consequently, they moved slowly through the app and it was easy to consume an entire session simply testing view account balance and transfer funds. More sessions were needed to test add payee, confirm payment was successful, dispute payment, etc

Effort Required to Produce Test Materials - Are you testing an existing app, paper prototypes, or an interactive prototype? The more you need to produce, obviously, the bigger the effort. In my experience, this is the largest variable that affects project duration.

Difficulty of Recruiting Participants - I'm currently testing an app (more LeanUX than usability but still relevant to this question) that has ~200 possible users nationwide. Each controls around $2 billion in spending annually. Getting their time has proven to be very difficult. Can anyone off the street be a test subject, or do you need to invest significant time recruiting test subjects with unique qualities?

Number of Personas - If you're testing 6 distinct personas, obviously this is much more time consuming than 2. It tends have a multiplying effect on the other aspects of testing - more personas often requires more screens in the prototype, more effort in recruiting and is an indicator of software complexity.

My Advice I can't tell you how to estimate the effort. But I can suggest an approach that's worked for me in the past. Borrow a trick from Agile software development and timebox. Tell management that you want to test 1 persona using simple paper prototypes of one particular feature. Test five participants and promise you'll get it done in 40 hours (this assumes that recruiting participants is relatively easy). Odds are you'll get the task done pretty close to the limit given to you. This allows you to return to management with a) some interesting results on how the app can be improved and b) a much better handle on what it takes to do usability testing with your team and your users on your app. This tends to make subsequent decisions easier and increases comfort levels with approving larger testing efforts.

Answer 2

For the most part it appears that the numbers you are quoting are still pretty valid. The biggest variable is going to be based on what you are testing. If you are testing existing functionality you will save a lot of time because you wont need to develop prototypes or mock-ups. For new functionality your numbers will be much closer to the ones you quoted.

If you are looking into decreasing your numbers you might want to look for a user testing service like Verifyapp. My organization has recently started using it for smaller user testing scenarios and it seems to be working out. The prices are reasonable and tests are very easy to setup.

Mind you this might not be the best software for your user testing. There is quiet a few options out there if you are looking for some automated solutions. These solutions normally speed up your setup and analysis phases.

Answer 3

You might be interested in this article by Jeff Sauro explaining the KLM model and how it can be used to provide an estimate for task completion time. The model is based on several key assumptions about user behaviour and competency (which is also based on previous research), but it does provide you with a basis to construct your own model and tweak some of the parameters. It can also be adopted for other interface types and interactions if you understand the fundamental principles behind it.