How to validate large-scale IA without skewing data? Crowd-sourced techniques?

Question

I am pitching for a job that requires validation of the information architecture of a very large health and social care website of around 6500 pages.

Here is my approach:

1. Analyse existing research: the domain is health and social care - I have no doubt there will be a wealth of research data for various demographics that will inform what sections are particularly important
2. Narrow focus for testing: 6500 pages is impossible but, in light of research from phase 1, we can narrow the focus to x key sections
3. Perform card sort with x users to build prototypical IA
4. (Retest - optional)

Concerns: the site is aimed at a very broad demographic. To do the card sort properly will require many users to yield anything statistically significant but the client has limited budget so this may not be possible.

So here is the question:

For a large and complex information architecture, with many different types of users, how does one validate the IA without skewing the data because of small sample size? Are there any cheap, large scale crowd-sourced techniques you know?

Answer 1

Basically the validation for large IA is similar as in other situations. Do a closed card sort.

Other comments on your problem: Check the the article Scaling the card sort method to over 500 items: restructuring the Google AdWords help center by Nakhimovsky et al for ideas on how to conduct card sorts with a lot of items. In essence you have to get the number of items to sort to manageable number.

As to the broadness of the demographic, I think you have two reasonable options: either do one card sort with a lot of users and use the results as basis for IA or do one for each of the major user groups and use those results as the basis of design. If I thought that doing separate sorts for each group would provide extra information that would actually get used in the design process, I would do that. If not, I would do just one sort with more users.

On number of users when card sorting: you would need roughly 15-20 users(according to Tullis & Wood, 2004) per user group to get reasonable results. I'd probably use one of the web based card sorting services, that way you can get reasonable number of users fairly cheaply as the compensation to users can be less as web based sorting is more convenient. The downside is that you don't get the qualitative data in the form of user comments and reasoning of the sort. The upside is that quantitative data should be comparable to paper based sort(Bussolon et al, 2006)

Anyway, that's of the top of my head.

Answer 2

Note: I represent Optimal Workshop, the company who makes the tools I propose below.

If your budget allows, I'd suggest you begin with a moderated card sort with a small number of participants (3-15). You will probably learn something about your card labels and content coverage in this process, so you can make any necessary adjustments to your cards at the completion of this stage. If the budget is tight, you could skip this step (or test with less people) and allocate more resources to user testing at a later stage.

Then perform a remote and unmoderated OPEN card sort using a tool such as OptimalSort with around 50 cards and as many participants as you need to get enough data to give you and your client confidence in your decisions.

If it is clear that certain user groups are only interested in certain content you could consider performing specific card sorts for specific groups of users. Card sorting will give you a good idea of your top level categories and clues for the lower levels. You can repeat card sorts at lower levels of content if you feel you don't have enough info to create the full tree at this stage.

Once you have formed your proposed/prototypical IA use a Tree Testing tool, such as Treejack, to validate your IA in a task based context. This is instead of a closed card sort. Tree Testing is described in this article by Dave O'Brien. You may need to do multiple Tree Tests to refine your IA before moving on to other stages of the project (so don't use up your whole database of potential study participants in the first hit). It could also be useful to segment your users and to test each segment with appropriate tasks against the same tree (IA).