How do you know, when a closed card sorting is valid (for website navigation)?

Question

I wonder if there is a way to tell, when the results of a closed card sorting test are statistically significant (e.g.: number of participants or a formular).

Background: I am working for a content heavy website (>1500 articles) and we want to improve our navigation.

In a 1.step we did an offline card sorting test and based on the results we came up with 6 Categories, which we now want to validate through the closed card sorting test. In this test, we want to test how well our 400 articles, fit into these 6 Categories.

Answer 1

In Measuring the User Experience by Tom Tullis and Bill Albert, they devote a section to the analysis of testing closed card sorts (Chapter 9 section 9.2.2).

They mention that the main thing you are interested in is how well the groups "pulled" the cards to them that you intend to belong to those groups. For example, the percentage of participants who put each card into each group.

The last column lists the winning group. What you hope to see are groups like Card#10 , with 92% of the participants putting it in that group. Ones that are more troubling are cases like Card#7, where 46% of the participants put it in Group A, but 37% put it in Group C - participants were very "split" in terms of deciding where that card belonged in this set of groups.

Data can also be analysed using hierarchical cluster analysis and MDS analysis.

Answer 2

One this to consider is the percentages of how consistently they were placed together and the naming conventions for that grouping.

If for 80% of your cards are 70% or more, skip the closed card sort and go for a tree test.

The mental model for grouping and finding are very different. The card sort is best used for understanding general groupings, finding good labels in their own words, and discovering where there are differences among them.

That data along with the content can help you draft the initial navigation to then do a tree test by asking participants real tasks and see where they click first and where they think the information will be.

This will allow you to rapidly iterate without the design to see how well the organization will be for most users.