Production Reaction Cards - too many?

Question

Microsoft Production Reaction Cards (doc)

The set includes 118 words which presumably you would reorganise, format and print off as individual "cards". I've never used them but I do see how the tool could be useful. My only concern is that there are too many words and it could make it unwieldy to deploy in user testing.

What are your experiences with using the MS PRCs? How did you present them to people and did you find that the list could probably be culled significantly? Can you break it down into groups and focus it at specific questions to elicit responses from that group of words? For example you could group the words relating to credibility and deploy them for questions relating to credibility: Trustworthy, Consistent, Dated, Inconsistent etc.

Answer 1

We find that asking people to sort through a stack of product reaction cards is an unnecessarily fiddly way to get satisfaction data. Instead, we print out the adjectives as a checklist and ask people to select the ones that apply. We've created an Excel spreadsheet that makes it easy to adapt the instrument to your own purposes and create a tag cloud of the results.

However, I'd re-iterate Jon's comment about using it as a probe. The best data comes from the post-test interview, where you ask people why they picked certain words.

Answer 2

We use a subset of these (with some minor modifications) for post-test exercises. There are a variety of ways of presenting them - one of the easiest we have found is to pre-prepare several print outs so the words are presented in a random order (different for different participants). Each participant then crosses out those they DO NOT think apply. We then have them do a prioritisation task with the ones remaining.

We are still working through factoring out some of the words but it is much more complex than you might at first think in order to produce a reliable test instrument (just read anything about standardised questionaires and you will see what I mean) - with the PRCs and other tools like them it is interesting to probe why people select any of the cards qualitatively - this can reveal some 'interesting' interpretations and is particularly important in tests will low numbers of participants where you can find there is actually a good level of overall consensus it is just that the participant chose to express it via the instrument differently (in this case the instrument is used more as more of a probe than an analytic device)).

For instance 'dated' is not always a bad thing... some people can take this to mean 'you put dates on your documents/posts which is really useful', sometimes dated means 'authoratitative' rather than 'old fashioned and out of date'

Answer 3

We use this routinely because generating word clouds in wordle is fantastic for presenting a very quick summary to senior stakeholders.

We've done it with index cards and with the Userfocus spreadsheet and I'm toying with the idea of building an electronic version to bolt onto the end of our remote unmoderated usability testing tool.

Whatever the mode of presentation, It's not a problem for us to reduce the number of words/cards because we're using the results as a very crude measure of subjective satisfaction. When we are designing against a set of core design principles I'll put representative words into the target list as a quick test to see if we're hitting the mark.

Cheers
R

Answer 4

For mojoleaf.com I cut the list of words down to around 70. Although it is still a lot it seems to be manageable based on initial feedback.

Answer 5

I'm a fan of the product reaction cards because of their ability to collect subjective data not always available through standard usability testing.

Unfortunately, a paper-and-pencil or paper-and-printer methodology is laborious.

A few of the problems are:

Creating the cards for every term in the list

Controlling for order effects in the presentation of the terms in the list

Recording the terms selected

Putting the selected terms into a form suitable for analysis

In my experience, the whole procedure, done manually, seemed unwieldy and not suited for a study with many participants or a quick turn around of analysis.

I chose to automate much of the procedure.

I wrote a PowerPoint-based VB program that:

reads the terms from a text file

creates one slide per term

adds a button to each slide that allows the use to indicate (yes/no) whether the term describes the product

creates a file with the list of terms chosen by the participant

The program allows us to:

update the terms in the list without have to create new slides, cards, or pieces of paper

randomize the order of presentation of the terms

quickly get the data into a form suitable for analysis

measure response time to determine if some terms are associated with longer or shorter response times than others

generate frequency counts for terms in the initial set and terms in the smaller set

A couple of other methodological points:

We told the participants to decide quickly because, in the absence of that instruction, they labored over each decision without changing the outcome of the decision.

Progress through the list was in one direction. In other words, the participant got one chance to decide for each term and could not revisit terms from earlier in the list.This is not the same as a manual sorting task in which the participant can move terms around indefinitely.

We narrowed the terms to a set of 80 or so.

Most participants finished in less than 2 minutes.

We then asked them to pick the 5 most descriptive and explain why they picked those terms. The really useful information from the participants was in the debrief when we asked them to tell us why they chose the 5 terms.

The analysis of the frequency with which terms are chosen or chosen together helps us decide whether terms are redundant and relevant. We would like each term to contribute new information rather than have several related terms that all mean the same thing to the participants.