Nonparametric vs. Parametric tests

Question

I work as a UX Researcher for a mobile gaming company.

We're discussing best practices in analysis, and my colleague suggested that we always use nonparametric tests (e.g., Mann-Whitney U rather than t, Kruskall-Wallace rather than ANOVA) because they do not suffer from the violation of assumptions that parametric tests do.

I've only ever used nonparametrics if I had to (e.g., the data were counts or ranks rather than means). The argument that one should always use nonparametrics doesn't sit well with me, but I certainly haven't done exhaustive research on the subject.

Are there any guidelines as to when we should use parametrics vs nonparametric tests?

Thank you!

Answer 1

I am surprised that there aren't more people responding to this question because there is definitely a need for more rigour around statistical analysis in usability testing.

I think you'll find some good resources from Jeff Sauro from MeasuringU, who seems to be one of the people that does a lot of consulting work in this area. I like most people, know little about statistics but at least I don't believe any stats I get without knowing how the figures were produced.

For me there is a more important thing compared to deciding the type of statistical analysis that you use to get your p-value - it should be about whether you've asked the right question in the first place, and whether your answer can be validated in more than one way. That's where the power of statistics come into the fore, being able to add some analysis and decision process to your decisions, but not be the reason for making those decisions.

Answer 2

In the case of applied research analysis, it's wise to avoid choosing a blanket approach. As @Michael Lai mentioned, your hypothesis / research question should drive your choice of statistical method and Jeff Sauro's blog is a good resource. (I'll avoid my tirade about the lack of real hypothesis testing in UX research for later because it sounds like you ARE testing hypotheses in your role.)

That said, there are some underlying assumptions about your data that dictate whether you should use parametric over nonparametric approaches. The rule of thumb that I was taught (in several stats courses) in graduate school is that if you have interval data as your dependent variable, use parametric tests. Like you mention, if you are measuring categorical, rank, or some forms of choice data as your outcome, then you should use a nonparametric test.

ANOVA specifically is extremely robust to violations of assumptions of normality and equal variance (the two that I feel we know the least about in UX studies, so are most likely to violate). Further, there are several types of ANOVA that were derived to allow more flexibility in research design and handle interval data that violate core assumptions (e.g. MANOVA, ANCOVA).

So, no. You should NOT just use nonparametric tests in UX research. Use the tool that's appropriate for the job.