I'm a fan of the product reaction cards because of their ability to collect subjective data not always available through standard usability testing.
Unfortunately, a paper-and-pencil or paper-and-printer methodology is laborious.
A few of the problems are:
Creating the cards for every term in the list
Controlling for order effects in the presentation of the terms in the list
Recording the terms selected
Putting the selected terms into a form suitable for analysis
In my experience, the whole procedure, done manually, seemed unwieldy and not suited for a study with many participants or a quick turn around of analysis.
I chose to automate much of the procedure.
I wrote a PowerPoint-based VB program that:
reads the terms from a text file
creates one slide per term
adds a button to each slide that allows the use to indicate (yes/no) whether the term describes the product
creates a file with the list of terms chosen by the participant
The program allows us to:
update the terms in the list without have to create new slides, cards, or pieces of paper
randomize the order of presentation of the terms
quickly get the data into a form suitable for analysis
measure response time to determine if some terms are associated with longer or shorter response times than others
generate frequency counts for terms in the initial set and terms in the smaller set
A couple of other methodological points:
We told the participants to decide quickly because, in the absence of that instruction, they labored over each decision without changing the outcome of the decision.
Progress through the list was in one direction. In other words, the participant got one chance to decide for each term and could not revisit terms from earlier in the list.This is not the same as a manual sorting task in which the participant can move terms around indefinitely.
We narrowed the terms to a set of 80 or so.
Most participants finished in less than 2 minutes.
We then asked them to pick the 5 most descriptive and explain why they picked those terms. The really useful information from the participants was in the debrief when we asked them to tell us why they chose the 5 terms.
The analysis of the frequency with which terms are chosen or chosen together helps us decide whether terms are redundant and relevant. We would like each term to contribute new information rather than have several related terms that all mean the same thing to the participants.