3

Instead of having one large subject pool of card sorting participants, has anyone ever tried segmenting their participants with a supplemental survey to define which persona they may fall into? If so, how did you go about analyzing the card sort data and segmenting the results for each of the persona groups?

(I have set up my study in conceptcodify.com to run the card sort and would like to direct them there from Qualtrics, in which I plan to ask them questions that can be used as a grouping variable.)

  • Welcome to the site, @Pedro. Interesting first question. A lot of the questions here tend to be about interface elements, so it's nice to see a question about methodologies phrased in a way that makes it concrete enough to answer. (You could probably still improve the phrasing so that it's not soliciting individual experience but instead asking about the best practices when combining these methods.) +1 for looking at ways to combine methodologies. – Graham Herrli Sep 2 '14 at 20:42
  • Thanks @3nafish, I hope to be use these forums more often as people here seem to respond fast and provide their perspective. I am a cognitive psychologist so I rely heavily on quant and qual research methodologies to direct content and design, you may be seeing more questions on such methods as I go through them! – Pedro Gutierrez Sep 3 '14 at 21:57
2

The question is valid, but it has somewhat of a 'backwards' scent.

Normally, you conduct a research (eg, card sorting) with some research goal in mind. The goal will account for the various personas. In other words, you recruit participants based on the fit criteria defined by the research goal which accounts for personas.

Consider for example a site (or on app) that has two main personas - N (for domain novice) and E (for domain experts). You also know that the site usage share between them is 20% for N, and 80% for E. For research validity sake, I'd argue you should have in your card sorting the same amount of participants type N and E (say 12 for each group). In the analysis stage, you separate the two groups - meaning you analyse each separately. Great discrepancies in the results between the two groups could suggest an audience based IA; but if such path is not to be taken, group E (80%) should have more weight on the combined results.

If you only segment your participants after the card sorting, you may run into some troubles - what if 20 participants belong to group N, and only 4 to group E? That's just to warn against such practice.

But I think the general principle to follow is that the primary persona group gets more weight compared to secondary ones (or any additional persona classification).

  • That makes perfect sense that the audiences should be defined prior to the study. The recruitment here is what is holding me back because I do not have a pool of participants in which I can categorize their user type. My study is being solicited to anyone who accesses the webpage, so I need to create a variable that I can use to categorize them with. However, it is the integration of Qualtrics and Conceptcodify.com data sets that I am struggling with because of the output format. Lastly, what do you mean by "...audience based IA"? thanks. – Pedro Gutierrez Sep 3 '14 at 21:48
  • I was thinking that sometimes you won't necessarily know your users because it is a new concept or application, and even though you have some assumptions it might not unnecessarily be based on good information so you need to be more open about what the groups might be as well. So sometimes doing post sort analysis might add another valuable insight into the data compared to pre sort grouping. – Michael Lai Sep 3 '14 at 22:20
  • @PedroGutierrez. If you already have your personas, you should be able to work out the questions that will match a participant to the persona. Your case is common, and you just have to match each participant to a persona group and analyse the results of each group separately. – Izhaki Sep 3 '14 at 22:32
  • 1
    As for "audience base IA"... say you do card sorting for the main navigation, and the novices and experts had noticeably different preferences (or terms); you can integrate an element to the interface letting users choose their group and then modify the interface accordingly. Most times, this is done by introducing a menu options (utility or main navigation) like 'Man'/'Women' or 'hobbyist'/'professional'. I hope I have answered your question. – Izhaki Sep 3 '14 at 22:34
1

Many of the online cardsorting application allow you to provide some questions that you can use to segment the users into different demographics, or allow you to pre-select responders based on demographics. So it is always standard practice for me to group users for the analysis rather than only relying on the overall results. However, it may not always be possible to do this for certain studies, especially if you have very small numbers.

I can see the reason for segmenting based on age, especially if the effect of computer literacy might play a part in the time taken or outliers in certain aspects of usability. Gender is know to have certain biases in design preferences and aesthetics too.

However, in terms of a more in-depth analysis about the users so you can derive a persona, there might be some issues to consider:

  • How much should you read into the results because it is hard to know how representative the participants are of the user population?
  • How much does the additional survey impact on the reliability and quality of the responses, given that people don't want to spend much time doing surveys and questionnaires?
  • How much are you sacrificing in terms of the accuracy of the results once you split the users into even smaller groups?

I do believe that these online cardsorting tools should come with some standard questions that can help define or break the users up into meaningful groups, but first we might need to come to an agreement about what they are.

  • Michael, I agree with you entirely that user types should be defined, however that is something that may be unique to the study. If there were a card sorting program that allowed you to customize demographic questions that would be ideal. Maybe there is, but since I am a starting freelancer I can't help but go with the cost effective solutions! – Pedro Gutierrez Sep 3 '14 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.