Any value to gathering qualitative data from card sorts?

Question

One of the best tools I've found for card sort experiments seems to be websort, mainly because you can just send out a link for participants to go through the exercise at their own convenience.

However, at the end of the day this method just returns limited quantitative data, and prevents the researcher from exploring another entirely separate dimension of information. I've never actually performed a card sort, so I don't know if performing the experiment in person is really worth the time. But at the end of other types of research sessions, I like to get inside the participant's head.

Have you found that gathering qualitative data from card sort experiments is worth the time? Based on the nature of card sorting, I expect this would generally need to be more involved than a few follow-up questions that could be included in a non-proctored experiment.

Edit: For the bounty, I'm looking for a why-or-why-not response. If you feel that qualitative data is important, bonus points for specifying what type of information should be gathered and/or which questions should be asked.

Answer 1

Yes, qualitative data from card sorts can be valuable as a knowledge elicitation technique, especially when you're working in domains outside your expertise. The main benefit is that in addition to the categories, you get to know a bit more about the criteria that your participants are using and their thinking process. (Although if you've never done a card sort before, you might be surprised how much you can learn from a standard computer based card sort.)

If working with one participant at a time, use normal think-aloud instructions. Do a warm-up exercise if necessary. Alternatively, you could pair up your participants and have them talk to each other, or use a larger group.

• Encourage to make a pile for 'other' or 'don't know this item'. That way the sort will be focused on items that the participants have something to say about. You can always revisit the rest pile at a later stage.

• Ask why questions (that is, if they don't talk enough on their own). Why does this item belong in pile A? With really strong participants such as true domain experts you can even ask 'why not' without influencing the results (for example when asking a medical doctor to categorize symptoms).

• Often, during the sort people will change their mind. Add a new category, change a label, or move an item. This is where it gets really interesting. What was wrong, why is this better?

• Have some blank cards ready, in case a participant can't choose between two or more piles because he feels the item can belong to all of them. In that case you can duplicate the card.

• When the sort is complete, take a look at the results. Does your participant feel that any items were missing? If that's the case, it's easy to create a few extra cards and at them to the relevant piles.

• After the initial sort, you can move up or down into the hierarchy. Ask to split up large categories into subcategories. And ask to group multiple categories together into supercategories. You can strongly encourage splitting up by taking 3 random items, and asking which 2 are most similar (the other one will go into a new category, taking all items that belong there too with it).

• Some members of a category are more central than others. You can ask which items are the best, most representative, example(s) for each category.

• Ask questions about the chosen labels. What makes them a good description? Alternative wordings?

• Ask questions about the participant. How familiar are they with the items? Do they consider themselves an expert in the domain? If so, did that influence their sorting?

With this type of card sorting the final groupings are less important than the thoughts during the process. Your questions may influence the participant's behavior, so I wouldn't mix the results from such sessions with those where participants work on their own.

Answer 2

Yes but it should be done in a physical setting where participants can think out loud and quickly be able to create new items without worrying about pressing the right/wrong button.

Some of the best findings you get from users cannot be obtained via simple quantification.

Answer 3

I'm curious, does this program tell you how long a person took to sort a particular card? or if they placed it in one group first and then the other? Does it let them keep any cards unsorted?

I'm also a novice to card sorting so I would like to add a sub question: Do people physically arrange cards or card groups in particular ways while or after sorting the cards? Would you as the person conducting the card sort care about the distances between the card groups or their location on the table?

Answer 4

In my opinion, especially performing an open card sort where participants can define their own categories can yield very helpful qualitative results.

The reason: often we have a lot of items that have to be displayed on our website/in our application and we may have a mental model already which suggests a sensible categorization scheme, but the categories our users come up with frequently have very little in common with our own.

Since we are not our users (and neither are our bosses), it's very important to figure out which categorization scheme is likely to work best for our target users, and card sorting can be a very valuable tool for this.

Answer 5

You should ask youself the big why...

We have dosins of dosins of methods you can use in your usability engineering, but everything you do must have a purpose. You don't just spend a few hours (and worse, a few users) if you don't need that information.

Obiously you can learn a lot when you work with users, and that is the main reason for gathering qualitative data during card sorting. Insight. But if the main purpose is to confirm an information hierarky that you already have worked a lot with, you don't need the qualitative information. You can do with some quantitative statistics.