User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

One of the best tools I've found for card sort experiments seems to be websort, mainly because you can just send out a link for participants to go through the exercise at their own convenience.

However, at the end of the day this method just returns limited quantitative data, and prevents the researcher from exploring another entirely separate dimension of information. I've never actually performed a card sort, so I don't know if performing the experiment in person is really worth the time. But at the end of other types of research sessions, I like to get inside the participant's head.

Have you found that gathering qualitative data from card sort experiments is worth the time? Based on the nature of card sorting, I expect this would generally need to be more involved than a few follow-up questions that could be included in a non-proctored experiment.

Edit: For the bounty, I'm looking for a why-or-why-not response. If you feel that qualitative data is important, bonus points for specifying what type of information should be gathered and/or which questions should be asked.

share|improve this question
up vote 4 down vote accepted

Yes, qualitative data from card sorts can be valuable as a knowledge elicitation technique, especially when you're working in domains outside your expertise. The main benefit is that in addition to the categories, you get to know a bit more about the criteria that your participants are using and their thinking process. (Although if you've never done a card sort before, you might be surprised how much you can learn from a standard computer based card sort.)

If working with one participant at a time, use normal think-aloud instructions. Do a warm-up exercise if necessary. Alternatively, you could pair up your participants and have them talk to each other, or use a larger group.

  • Encourage to make a pile for 'other' or 'don't know this item'. That way the sort will be focused on items that the participants have something to say about. You can always revisit the rest pile at a later stage.

  • Ask why questions (that is, if they don't talk enough on their own). Why does this item belong in pile A? With really strong participants such as true domain experts you can even ask 'why not' without influencing the results (for example when asking a medical doctor to categorize symptoms).

  • Often, during the sort people will change their mind. Add a new category, change a label, or move an item. This is where it gets really interesting. What was wrong, why is this better?

  • Have some blank cards ready, in case a participant can't choose between two or more piles because he feels the item can belong to all of them. In that case you can duplicate the card.

  • When the sort is complete, take a look at the results. Does your participant feel that any items were missing? If that's the case, it's easy to create a few extra cards and at them to the relevant piles.

  • After the initial sort, you can move up or down into the hierarchy. Ask to split up large categories into subcategories. And ask to group multiple categories together into supercategories. You can strongly encourage splitting up by taking 3 random items, and asking which 2 are most similar (the other one will go into a new category, taking all items that belong there too with it).

  • Some members of a category are more central than others. You can ask which items are the best, most representative, example(s) for each category.

  • Ask questions about the chosen labels. What makes them a good description? Alternative wordings?

  • Ask questions about the participant. How familiar are they with the items? Do they consider themselves an expert in the domain? If so, did that influence their sorting?

With this type of card sorting the final groupings are less important than the thoughts during the process. Your questions may influence the participant's behavior, so I wouldn't mix the results from such sessions with those where participants work on their own.

share|improve this answer

Yes but it should be done in a physical setting where participants can think out loud and quickly be able to create new items without worrying about pressing the right/wrong button.

Some of the best findings you get from users cannot be obtained via simple quantification.

share|improve this answer
Thanks for your feedback. Are you discouraging the use of all card sort programs, including those for administering the test in person (and using a more traditional method like note cards instead)? – Baa Apr 25 '11 at 16:22
+1 Could work remote as well though, e.g. with Skype or something similar used in parallel (see also ‘Remote Research’ by Nate Bolt & Tony Tulathimutte). – Sascha Brossmann Apr 25 '11 at 22:26
I'm looking for more specific information as to why qualitative data is valuable for card sorts. I feel that both open card sorts and close card sorts have a time and place to be used. I would not really consider an open sort (the ability to create new items) to be qualitative data, per se. – Baa May 1 '11 at 17:09

I'm curious, does this program tell you how long a person took to sort a particular card? or if they placed it in one group first and then the other? Does it let them keep any cards unsorted?

I'm also a novice to card sorting so I would like to add a sub question: Do people physically arrange cards or card groups in particular ways while or after sorting the cards? Would you as the person conducting the card sort care about the distances between the card groups or their location on the table?

share|improve this answer
Hi Viraj, for websort all of the cards are available at once. The user can sort them in any order. I think the paid version gives information like how long the user took to complete the sort, and perhaps how many times each card was moved. As for your second question, there are many different ways to perform a card sort. Sometimes there are no groups to start out with, and the user creates their own groups. My understanding is that physical distance does not matter as much, and these experiments probably take a lot of space when a lot of cards are being used. – Baa May 1 '11 at 17:26

In my opinion, especially performing an open card sort where participants can define their own categories can yield very helpful qualitative results.

The reason: often we have a lot of items that have to be displayed on our website/in our application and we may have a mental model already which suggests a sensible categorization scheme, but the categories our users come up with frequently have very little in common with our own.

Since we are not our users (and neither are our bosses), it's very important to figure out which categorization scheme is likely to work best for our target users, and card sorting can be a very valuable tool for this.

share|improve this answer
Thanks Jan. Based on the feedback I'm getting it seems like there is a consensus for using open card sorts when interested in collecting qualitative data. Makes sense. – Baa May 3 '11 at 0:04

You should ask youself the big why...

We have dosins of dosins of methods you can use in your usability engineering, but everything you do must have a purpose. You don't just spend a few hours (and worse, a few users) if you don't need that information.

Obiously you can learn a lot when you work with users, and that is the main reason for gathering qualitative data during card sorting. Insight. But if the main purpose is to confirm an information hierarky that you already have worked a lot with, you don't need the qualitative information. You can do with some quantitative statistics.

share|improve this answer
Thanks, Jorn. Good point - I guess should just use online card sorts when I want to perform a closed sort to validate an existing architecture, but I should consider in-person sessions for open card sorts that pertain to newer projects. – Baa May 3 '11 at 0:08

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.