I am conducting paper-prototype testing and manage to collect data about completion rates and satisfaction scores. However, I am finding it difficult to calculate the task times. Is there another metric apart from response time I can possibly use to evaluate paper prototypes?
I like earlier responses, but in addition, I would point out that if users were not aware that time was an issue, you will have so much variability in your data that unless the effect is huge, it will not be significant.
In the future, you can use systems like Verify (verify.com) allow you to quantify completion rates and task times on individual screen mockups.
When it comes to additional quantifications, I would definitely do subjective rating, meaning this is important to the user. You could do it in absolute, or comparative to what they would otherwise use. You start by defining what are the important attributes of the experience (speed, effectiveness, trust, enjoyment, etc.) and ask people to rate the experience for those top 3-4 attributes.
Sounds like you're trying to gather quantitative data from what's essentially a qualitative research method. I've only ever gathered usability information from usability tests. I'd expect low-fi prototypes to give even less-precise quantitative data than a finished system.
But you can discover some very useful qualitative data with low-fi prototypes. Mainly, you can discover problem areas and use your observations to improve your early designs. And, since it's a paper prototype, which allows you to easily create multiple versions, you can test them against each other.
If you're really interested in completion times, try a KLM-GOMS model, which provides exactly that, with surprising accuracy.
I would ask respondents to estimate ease of completion for task (such as 1-very difficult -> 7-very easy) or ask them to map how easy they'd think it would be to do within a real app, if that's part of your goal.
Maybe like you, I've found too many potential distractors with paper-prototypes to make stopwatch time useful.
You could count:
- Number of times user has to ask for clarification, grouped by task
- Number of times user has to ask for clarification, grouped by element
- Number of times user "presses" or interacts with an element, only to realize it doesn't do what expected
- Count the number of times users accomplish task X on the first attempt.
Asking users for their feedback on how hard something is can be useful, but users are not especially accurate in what they say, which is why data collected by observing them can be more useful.