I am planning to run some one-to-one think aloud task based usability tests on a wireframe prototype. At the end of the test I would like to ask the users to take part in an easy exercise (something like card sorting or similar) where they would be asked to prioritise and comment on the importance of upcoming features for the service.
Can you recommend some kind of popular method/exercise for this purpose?
I don’t think it will work well to provide the user with a description of each upcoming feature and ask them to rate or sort them. That requires users to try to imagine what the upcoming feature would be like in their life once it’s implemented. That’s much harder than a usability test, where users can try out exactly how a feature is implemented and give you feedback. For upcoming features, you can’t control what the users are going to imagine (if they can imagine it), so the results tend to be unreliable. You can try to explain the feature in detail (verbally, in text, or graphically) but that takes a long time, and it requires that the user hold a lot in his/her head when trying to do the ratings.
Instead, derive a set of questions about the means and ends of each feature from the user’s perspective. For each feature, you want to find out
The frequency that users have the goal that the feature achieves for them.
The importance of achieving the goal.
The degree the users have the resources to use the feature as it’s planned.
The degree the feature as planned may annoy or frustrate the user.
Combine these and you have the value of the feature (and priority, all other things being equal).
For example, say one feature for your social site is to send the user a text message whenever someone posts a response to a user posting. The goal of the feature is to eliminate the need for the users to constantly check back on the social site to see if there’s response. So the specify questions become something like:
How often do users check back at social site just to see if there’re responses?
How often do users find themselves wishing they knew if they had response yet?
How often do they have their phone on them when they’re wishing such things?
How disruptive or annoying is it to get and check text messages?
You may need more than four questions.
I’d take advantage of the usability test to do a mini contextual inquiry to answer these questions (or maybe “semi-contextual” inquiry if it’s in a usability lab rather than on the user’s home turf). Close the prototype and open the service as it exists now (or let the user open the competitor's service they’re currently using), and engage the user in a brief open-ended discussion on how they are currently using the service in order to answer the above questions. At the end of the discussion, try to get the user to put numbers down for each of the specific questions so you can compare the results of the upcoming features against each other. The hard part may be coming up with specific but comparable questions for each feature.
We did this at work.
Things you need to keep in mind when it comes down to card sorting or anything similar is to put the least amount of information possible and see where the user ends up. What you're trying to do is see if the user can figure it out on their own without any extra help (unless otherwise needed or instructed).
What we did was get all of our navigation parent pages and all the sub navigation pages and said, which sub navigation pages match up with the parent pages? It was really interesting to see how confusing in some areas or how clear cut in other areas our navigation was in terms of wording. We used this site which allowed for card sorting to take place online (not necessarily in person, but you can get them to do it after you get them to test the wireframe): http://www.optimalworkshop.com/ .
You can do this in various other methods like create a table on a piece of paper and cut out other colored pieces of paper with what you're trying to sort or have them prioritize and let them put the cut pieces of paper where they think they should be in the paper table. I don't really like this method because not only is it time consuming for both parties, but then you also have to get all the data yourself. Use the site given above, it's much easier (or other sites that do the same thing, really doesn't matter).
I hope this helped!
