4

I am trying to find examples of user behaviour research that combines both quantitative and qualitative data. It seems to me that most of the research leans heavily or exclusively towards one approach or the other. So you often see a large-scale survey study drawing some very specific conclusions, or heat-map or eye tracking studies that make some very general conclusions. The extent to which you can interpolate or extrapolate these results surely depends on if you are able to link them to a specific context. Considering that there are obvious advantages and disadvantages to each approach, what is the major concern with researcher running studies to collect both types of information? I don't by the argument of cost or time, because if the information being collected is not accurate or cannot be put into context correctly then it is a much bigger waste of time.

One of the key take-home messages for UX practitioners from Comparative Usability Evaluation 8 is to:

"Combine qualitative and quantitative findings in your report. Present what happened and support it with why it happened."

Because by applying qualitative and quantitative research methods you are able to treat a problem by linking the symptoms (what happened) to the root cause of the problem (why it happened).

Why isn't quantitative and qualitative data collected at the same time in more UX research/studies?

Improve this question

4 Answers 4

Reset to default
7

You’ll find a taxonomy and descriptions of different ways of integrating qualitative and quantitative methods in the “Mixed Methods Procedures” chapter in John Creswell’s (2013) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (SAGE Publications). The chapter includes examples of actual studies. The book is intended for social science field research, but the chapter can be applied to UX work too. For example, Luke Swartz’s thesis Why People Hate the Paperclip: Labels, Appearance, Behavior, and Social Responses to User Interface Agents is an example of a “sequential exploratory strategy.”

To summarize, you integrate the qualitative and quantitative results by:

  • Using the qualitative data to explain what you found in the quantitative data.

  • Using the qualitative data to decide what to manipulate and measure quantitatively.

  • Use one method to validate the results of the other (where one makes up for the weakness of the other).

  • Use whichever method is most suitable for each component of your theory you’re testing.

Qualitative and quantitative methods have different needs, so they are often conducted separately, almost as if they were separate studies. They often use different samples or sampling procedures. However, you can do mixed-methods in a single round of usability testing:

  • You encourage the user to think aloud, which (I’ve always assumed) has little impact on the relative quantitative performance (e.g., time to complete task, number of errors, eye-gaze durations). To get accurate quantitative results, avoid interrupting the user, perhaps doing so only if they get stuck for standard period of time.

  • At the end of the test, you can use a quantitative survey (e.g., SUS). After that you conduct a qualitative interview/debrief (which may involve going back into the app to discuss what happened at various places).

  • You perform qualitative analysis of the video tapes, the eye-tracking data (which may also be analyzed quantitatively), and interview results. Because qualitative analysis is time-consuming per user while quantitative methods need a relatively large sample size, you may want to perform qualitative analysis on subset of your users.

  • You integrate the results from the methods using one or more of Creswell’s strategies.

Improve this answer
4

I can only speak from personal experience, but I often run mixed-methods research that combines surveys and qualitative data. This gives you complimentary data such as "Quantitatively, users preferred Site A to Site B. The focus group explains some of the reasons why they preferred Site A."

Improve this answer
7
  • Is this UX work within an organization or some consulting service that you provide for companies? I find it difficult enough to get the time and resources (i.e. money and infrastructure) to do one let alone both.
    – Michael Lai ♦
    Sep 5, 2013 at 2:01
  • Yes, I work for UserTesting.com and the tools are handled mostly by the site infrastructure. However, there are many ways you could approximate this kind of research-- for example, you could release a survey via SurveyMonkey, MTurk, etc., and then contact a few willing participants to do an interview or usability session. The results of the survey could dictate what you focus on during the in-person part.
    – BlindRorschach
    Sep 5, 2013 at 2:07
  • That's definitely one way to go about it, but I think there's more value in doing something in parallel so that you can rule out certain variability in the results. It may not necessarily make sense to look at matched result pairs, but matching overall qualitative and quantitative results would be more accurate than looking at either by itself.
    – Michael Lai ♦
    Sep 5, 2013 at 2:11
  • Can you expand upon what you mean by "doing something in parallel"? This is a question I'm always trying to address more effectively for my own clients. (Also, I'm curious as to how much time and cost factors in for you, as in the question you say those aren't excuses but in the comment you said you find it difficult to conduct both due to the resource drain)
    – BlindRorschach
    Sep 5, 2013 at 2:13
  • I am thinking about recording keystroke and mouse click information (or eye-tracking) while the user is carrying out a task and getting immediate feedback after each task and getting them to type in notes. In fact, many of the collaborate wireframing or mockup tools have similar functions built-in. This would be better than recording information for a bunch of tasks and then asking them to complete a long survey afterwards.
    – Michael Lai ♦
    Sep 5, 2013 at 2:33
2

Running qualitative and quantitative in parallel like you described above is an interesting idea. In my experience however the benefits from a quantitative study come fom the large sample sizes, tight margins of error, and generalizability. How might you account for trade off between large sample sizes in quantitative studies but the money and time it takes to run a qualitative studies with enough users to achieve the benefits of the large sample size?

Improve this answer
1
  • Your answer sounds more like a question, but it is a valid point that you are making. There are various articles and discussions on the number of participants you need to gain enough power or statistical significance in the findings, yet I think the value of qualitative studies is really to try and dissect or get to the bottom of trends or patterns identified in quantitative analysis. Conversely, when you conduct qualitative studies, it might also reveal user behaviours and interactions that are important to try and capture in larger scale quantitative analysis.
    – Michael Lai ♦
    Mar 17, 2015 at 22:01
1

Mixed methodologies are always the best, because you get both what happened and the why it happened. Check out these articles....

https://www.nomensa.com/blog/2015/ux-research-selecting-right-method

https://www.nngroup.com/articles/which-ux-research-methods/

You do not need qualitative data if you want to know, "what happened," and you do not need quantitative data if you only need to know, "why it happened." If you cannot observe your users in their context, you may need to use methodologies that can be implemented remotely.

If you are in the pre-design phase, you want to use qualitative methodology to understand what the product is. If you have launched, you want to implement quantitative research to know how your product is used and how often. E.g. Eye tracking or google analytics.

Responding to Lucas, quantitative and qualitative can both be expensive. E.g. Card sorting can take a long time to organize data and make accurate conclusions. All research requires large sample sizes to achieve validity.

However, if you are a startup without a lot of cash, you should use qualitative research to create mental models and best guesses. You can perform heuristic evaluations on similar products that are already on the market.

Young, Indi. 2008. Mental Models: Aligning Design Strategy with Human Behavior. New York: Rosenfeld Media.

Young, Indi. 2008. Mental Models: Aligning Design Strategy with Human Behavior. New York: Rosenfeld Media.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.