I've recently piloted an open card sort, and have had some feedback suggesting that respondents should be told that they can add a 'Don't know' group if they're not sure where to put cards. I'm reluctant to do so as I feel this would encourage respondents to put more cards into the 'Don't know' group than they would otherwise, and getting sufficient responses is unlikely to be an issue (I'm running it online, within a large company, with an incentive). Is there any research for or against 'Don't know' groups in card sorts?
I've just done a little quick and dirty analysis on the OptimalSort database for you.
Eyeballing some of the instructions messages suggests that most studies are not prompting the use of a "don't know" group. So I've counted the number of participants who did anyway using the following criteria:
Please forgive me if you can't read this, and please forgive the terrible regular expression if you can! You can see the list of partial words it is matching anyway (so it will also match "Miscellaneous", for example).
Using this criteria it seems that ~6.2% of OptimalSort participants are creating some variant of a "don't know" group, mostly of their own accord.
So based on this preliminary and far-from-perfect research I would tend to say you don't need to suggest it. If the participant doesn't know what to do with a card they will tell you.
I'd be more than happy to look a little deeper if it's useful to you.
It's a question of costs and risks. Which is worse: users getting an incentive without really trying, or false data being taken during research and becoming misread as "official user feedback"? I'd suggest the latter. Forcing categorisation means participants create weak categories just for the sake of including miscellaneous items. That means weak silos that don't make sense, undermining the whole exercise.
It's much, much better to come out of a test with an awareness that users don't have strong preexisting categories than an arbitrary navigation solution that doesn't provide insights for future projects.
Would it be a bad idea to allow them to define groups instead of using the Don't know group? This way, you get them to help with the categorization of cards and then you can use newly created groups to help other users who don't know what how to group their cards.
'Don't know' is fine but I would imagine that someone will have to step in behind someone making that kind of a decision and then add them to groups. The only downside with offering the user an ability to define groups is that you may end up with groups that can't be managed later on.
I also agree with you that if the user didn't want to take the time to search then they would just classify everything as a Don't know making the whole point of grouping the cards futile.
We had a similar issue with a client I used to work for and what we did was offer the users the ability to type in what they thought the group was and then simply stored this non-system group in the back-end. Other users of the system got to see this grouping but didn't necessarily have to choose it. When an admin went into the category management they could approve the category and then it became a system category.
The point of a card sort is to figure out what mental modal your users have of the information you're presenting, before they see the site. The main thing it lets you do is to avoid the frustration of not finding the foo section under the bar section, when that's obviously where it should be.
If all your users put foo in 'don't know' then that tells you something important: that they would probably use other means to find it (like search or an instruction page), or that they won't mind clicking around a little bit. They don't have expectations of where it should be, so they'll be more patient.
This means that omitting the 'don't know' class is poisoning your data. What the card sort gives you is the places where data should definitely be. Without the 'don't know', you'll end up putting lots of other items under bar as well, making foo more difficult to find, without real benefit.
If you're still worried about the items in the 'don't know' category, split your respondent group in two. Let the first do an open sort on all items. And let the second do an open sort on only the items that were put in 'don't know' by the first. Make sure you have a 'don't know' in both cases.