Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I am trying to find examples of user behaviour research that combines both quantitative and qualitative data. It seems to me that most of the research leans heavily or exclusively towards one approach or the other. So you often see a large-scale survey study drawing some very specific conclusions, or heat-map or eye tracking studies that make some very general conclusions. The extent to which you can interpolate or extrapolate these results surely depends on if you are able to link them to a specific context. Considering that there are obvious advantages and disadvantages to each approach, what is the major concern with researcher running studies to collect both types of information? I don't by the argument of cost or time, because if the information being collected is not accurate or cannot be put into context correctly then it is a much bigger waste of time.

What are examples of studies that you have read or conducted that collects and analyzes both quantitative and qualitative data?

share|improve this question
add comment

2 Answers

up vote 5 down vote accepted

You’ll find a taxonomy and descriptions of different ways of integrating qualitative and quantitative methods in the “Mixed Methods Procedures” chapter in John Creswell’s (2013) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (SAGE Publications). The chapter includes examples of actual studies. The book is intended for social science field research, but the chapter can be applied to UX work too. For example, Luke Swartz’s thesis Why People Hate the Paperclip: Labels, Appearance, Behavior, and Social Responses to User Interface Agents is an example of a “sequential exploratory strategy.”

To summarize, you integrate the qualitative and quantitative results by:

  • Using the qualitative data to explain what you found in the quantitative data.

  • Using the qualitative data to decide what to manipulate and measure quantitatively.

  • Use one method to validate the results of the other (where one makes up for the weakness of the other).

  • Use whichever method is most suitable for each component of your theory you’re testing.

Qualitative and quantitative methods have different needs, so they are often conducted separately, almost as if they were separate studies. They often use different samples or sampling procedures. However, you can do mixed-methods in a single round of usability testing:

  • You encourage the user to think aloud, which (I’ve always assumed) has little impact on the relative quantitative performance (e.g., time to complete task, number of errors, eye-gaze durations). To get accurate quantitative results, avoid interrupting the user, perhaps doing so only if they get stuck for standard period of time.

  • At the end of the test, you can use a quantitative survey (e.g., SUS). After that you conduct a qualitative interview/debrief (which may involve going back into the app to discuss what happened at various places).

  • You perform qualitative analysis of the video tapes, the eye-tracking data (which may also be analyzed quantitatively), and interview results. Because qualitative analysis is time-consuming per user while quantitative methods need a relatively large sample size, you may want to perform qualitative analysis on subset of your users.

  • You integrate the results from the methods using one or more of Creswell’s strategies.

share|improve this answer
add comment

I can only speak from personal experience, but I often run mixed-methods research that combines surveys and qualitative data. This gives you complimentary data such as "Quantitatively, users preferred Site A to Site B. The focus group explains some of the reasons why they preferred Site A."

share|improve this answer
    
Is this UX work within an organization or some consulting service that you provide for companies? I find it difficult enough to get the time and resources (i.e. money and infrastructure) to do one let alone both. –  Michael Lai Sep 5 '13 at 2:01
    
Yes, I work for UserTesting.com and the tools are handled mostly by the site infrastructure. However, there are many ways you could approximate this kind of research-- for example, you could release a survey via SurveyMonkey, MTurk, etc., and then contact a few willing participants to do an interview or usability session. The results of the survey could dictate what you focus on during the in-person part. –  BlindRorschach Sep 5 '13 at 2:07
    
That's definitely one way to go about it, but I think there's more value in doing something in parallel so that you can rule out certain variability in the results. It may not necessarily make sense to look at matched result pairs, but matching overall qualitative and quantitative results would be more accurate than looking at either by itself. –  Michael Lai Sep 5 '13 at 2:11
    
Can you expand upon what you mean by "doing something in parallel"? This is a question I'm always trying to address more effectively for my own clients. (Also, I'm curious as to how much time and cost factors in for you, as in the question you say those aren't excuses but in the comment you said you find it difficult to conduct both due to the resource drain) –  BlindRorschach Sep 5 '13 at 2:13
    
I am thinking about recording keystroke and mouse click information (or eye-tracking) while the user is carrying out a task and getting immediate feedback after each task and getting them to type in notes. In fact, many of the collaborate wireframing or mockup tools have similar functions built-in. This would be better than recording information for a bunch of tasks and then asking them to complete a long survey afterwards. –  Michael Lai Sep 5 '13 at 2:33
show 2 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.