6

I work as a UX Researcher for a mobile gaming company.

We're discussing best practices in analysis, and my colleague suggested that we always use nonparametric tests (e.g., Mann-Whitney U rather than t, Kruskall-Wallace rather than ANOVA) because they do not suffer from the violation of assumptions that parametric tests do.

I've only ever used nonparametrics if I had to (e.g., the data were counts or ranks rather than means). The argument that one should always use nonparametrics doesn't sit well with me, but I certainly haven't done exhaustive research on the subject.

Are there any guidelines as to when we should use parametrics vs nonparametric tests?

Thank you!

  • I'm not sure this is type of discuss is on topic for UX.SE. But nonparametric tests are less powerful. Your first step should be to be visually inspect the data to see how it's distributed, then you decide what tests to use. – nightning Apr 15 '16 at 19:52
  • I think I need help understanding this forum! I've only posted three questions, and they've all been inappropriate for UX.SE. It might be that this isn't the community I'm looking for, or that I'm using it wrong. What am I doing wrong? Thank you! – SamuraiUX Apr 15 '16 at 21:24
  • I suggest taking the TOUR to get an better understanding. And your first lesson - Stack Exchange is not a forum. It's a Q&A site. – JonW Apr 15 '16 at 22:17
  • 1
    I agree that there is a correct(-ish) answer, and an "-ish" answer is perfectly fine here. I think there is an edit that can be done to that would pass through moderation on a re-open vote! Right now you're asking for opinion, which might have been why it was put on hold. Another idea for rephrasing the question could be to think of it in terms of a teacher to a class. The teacher isn't going to ask for open opinion, they're going to ask for the pros/cons - there is still flexibility in the correctness of an answer, but it frames the question towards a goal versus an open discussion. – Evil Closet Monkey Apr 18 '16 at 14:42
  • 1
    Voted to close. I think that this is actually a good question and it SHOULD be on-topic, but it seems like we don't get many people with a professional-level grasp of statistics on this site (myself included, unfortunately). It is definitely a BETTER fit on stats.stackexchange.com, I think that the best thing is to migrate there. – Vitaly Mijiritsky Apr 19 '16 at 6:10
1

I am surprised that there aren't more people responding to this question because there is definitely a need for more rigour around statistical analysis in usability testing.

I think you'll find some good resources from Jeff Sauro from MeasuringU, who seems to be one of the people that does a lot of consulting work in this area. I like most people, know little about statistics but at least I don't believe any stats I get without knowing how the figures were produced.

For me there is a more important thing compared to deciding the type of statistical analysis that you use to get your p-value - it should be about whether you've asked the right question in the first place, and whether your answer can be validated in more than one way. That's where the power of statistics come into the fore, being able to add some analysis and decision process to your decisions, but not be the reason for making those decisions.

0

In the case of applied research analysis, it's wise to avoid choosing a blanket approach. As @Michael Lai mentioned, your hypothesis / research question should drive your choice of statistical method and Jeff Sauro's blog is a good resource. (I'll avoid my tirade about the lack of real hypothesis testing in UX research for later because it sounds like you ARE testing hypotheses in your role.)

That said, there are some underlying assumptions about your data that dictate whether you should use parametric over nonparametric approaches. The rule of thumb that I was taught (in several stats courses) in graduate school is that if you have interval data as your dependent variable, use parametric tests. Like you mention, if you are measuring categorical, rank, or some forms of choice data as your outcome, then you should use a nonparametric test.

ANOVA specifically is extremely robust to violations of assumptions of normality and equal variance (the two that I feel we know the least about in UX studies, so are most likely to violate). Further, there are several types of ANOVA that were derived to allow more flexibility in research design and handle interval data that violate core assumptions (e.g. MANOVA, ANCOVA).

So, no. You should NOT just use nonparametric tests in UX research. Use the tool that's appropriate for the job.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.