Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I was setting up a usability test at my office where we have never done much formal usability testing. One of management's first questions was, "How long will this take?" I didn't have a good answer because I had been too busy thinking about writing scenarios, creating prototypes, recruiting participants, etc... to ever consider this.

Nielsen says a beginner could run a test in ~ 40 hours and I read an S&TC article (pg 7) saying that a good estimation formula was 10 hrs * # of testers * length of session. Both of these estimates seem reasonable but kind of out of date (Nielsen 1998, S&TC 2003).

Has anyone used these models before? Were they semi-accurate? Do you use a model you're quite fond of?

Note: I plan to track time spent on each task of preparing this usability test so I can use them as a base when planning future studies.

share|improve this question
    
What are you testing? Are you testing a brand new application> A few newly create site pages? Or added functionality of some kind? –  JeffH May 16 '12 at 13:32
    
# of testers * Session Length is obviously valid but the 10 hrs is probably more accurately split out into your standard prep/analysis time per trial, which can vary a lot depending on the shop and the test. –  Ben Brocka May 16 '12 at 13:35
    
@JeffH There are some new regulations requiring us to test some existing functionality. I expect that we will save some time in being able to use our existing software instead of a prototype/mockup. –  Andrew Shipe May 16 '12 at 13:45
    
@BenBrocka Splitting the 10 hours is definitely the challenge. I feel pretty comfortable with the process but there's a lot of bureaucratic weight to get moving as this is their first time with a serious usability test. Perhaps a high estimate and a healthy sized grain of salt is the best option here. –  Andrew Shipe May 16 '12 at 13:49
1  
Yeah, while it might sound bad on the face of it for your first usability test it's better that they're surprised it went so fast compared to your estimate as opposed to being angry it took longer. –  Ben Brocka May 16 '12 at 13:57

3 Answers 3

up vote 3 down vote accepted

In my experience, the effort invested in testing will vary considerably based on the complexity of the software you're testing, whether your test materials are prepared already, the difficulty of recruiting participants and the number of personas you feel you need to text.

Complexity of Software - I tested an Internet Bill Pay / Banking app in 1996 when that sort of transaction was complex for many users. Consequently, they moved slowly through the app and it was easy to consume an entire session simply testing view account balance and transfer funds. More sessions were needed to test add payee, confirm payment was successful, dispute payment, etc

Effort Required to Produce Test Materials - Are you testing an existing app, paper prototypes, or an interactive prototype? The more you need to produce, obviously, the bigger the effort. In my experience, this is the largest variable that affects project duration.

Difficulty of Recruiting Participants - I'm currently testing an app (more LeanUX than usability but still relevant to this question) that has ~200 possible users nationwide. Each controls around $2 billion in spending annually. Getting their time has proven to be very difficult. Can anyone off the street be a test subject, or do you need to invest significant time recruiting test subjects with unique qualities?

Number of Personas - If you're testing 6 distinct personas, obviously this is much more time consuming than 2. It tends have a multiplying effect on the other aspects of testing - more personas often requires more screens in the prototype, more effort in recruiting and is an indicator of software complexity.

My Advice I can't tell you how to estimate the effort. But I can suggest an approach that's worked for me in the past. Borrow a trick from Agile software development and timebox. Tell management that you want to test 1 persona using simple paper prototypes of one particular feature. Test five participants and promise you'll get it done in 40 hours (this assumes that recruiting participants is relatively easy). Odds are you'll get the task done pretty close to the limit given to you. This allows you to return to management with a) some interesting results on how the app can be improved and b) a much better handle on what it takes to do usability testing with your team and your users on your app. This tends to make subsequent decisions easier and increases comfort levels with approving larger testing efforts.

share|improve this answer

You might be interested in this article by Jeff Sauro explaining the KLM model and how it can be used to provide an estimate for task completion time. The model is based on several key assumptions about user behaviour and competency (which is also based on previous research), but it does provide you with a basis to construct your own model and tweak some of the parameters. It can also be adopted for other interface types and interactions if you understand the fundamental principles behind it.

share|improve this answer

For the most part it appears that the numbers you are quoting are still pretty valid. The biggest variable is going to be based on what you are testing. If you are testing existing functionality you will save a lot of time because you wont need to develop prototypes or mock-ups. For new functionality your numbers will be much closer to the ones you quoted.

If you are looking into decreasing your numbers you might want to look for a user testing service like Verifyapp. My organization has recently started using it for smaller user testing scenarios and it seems to be working out. The prices are reasonable and tests are very easy to setup.

Mind you this might not be the best software for your user testing. There is quiet a few options out there if you are looking for some automated solutions. These solutions normally speed up your setup and analysis phases.

share|improve this answer
    
Verifyapp looks very slick. I'll definitely check it out. –  Andrew Shipe May 18 '12 at 13:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.