I have an engineering background and I work as an interaction designer at a website company. I started to work 3 months ago. I am new in this field.
To interpret analytics data and usability tests, what statistics knowledge do I need to learn? What should I focus on?
Beyond that, to understand how severe a usability problem is, what should I know in terms of statistics?
https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/
Why Statistics?
Once you get beyond graphs, averages, and percents, the bulk of statistics concerns answering the question, “Is this sample size big enough to convince me that what I see in the sample actually applies in general?” This branch of statistics is called “inferential statistics.”
It’s definitely worth knowing and doing inferential statistics so you can tell when you have convincing data that one UI alternative is better than the other, or what is the likely range of impact (best case to worse case) a problem could reasonably have. Using inferential statistics can tell you when you have a sufficient sample size to terminate a usability test or A-B test and go with what you have, for example. If you have limited control of the numbers of users in the test (maybe it’s restricted by the time available), it’s still helpful to know how much confidence to have in your resulting UI design choice or prioritization.
Even if you’re only doing qualitative usability testing (concern with what the usability problems are, not how frequent they are), a good understanding of the principles of inferential statistics and the issues of sample size is helpful for explaining to stakeholders why one user flailing in a usability test of five users is justification for changing the UI intended for many thousands of users.
What Statistics to Know?
I consider an introductory college-level course in statistics to be both necessary and adequate for an interaction designer. For interpreting usability test results and data analytics, the following inferential statistical procedures cover most situations:
Usability data often have small sample sizes and/or very skewed results (e.g., small proportion of conversions), so you also should also be knowledgeable on
All the above are typically covered in an intro textbook (except maybe one-way ANOVAs, which some consider intermediate-level statistics).
What Statistics are Not Necessary?
Very rarely does usability work involve multi-way tests (e.g., all combinations of multiple UI variables), or true multivariate analyses (e.g., measuring the combined impact of a UI on multiple metrics). What some in UX call a “multivariate” analysis is really just a case of more than two UI alternatives being compared, which should be handled with a one-way ANOVA or multi-column Chi-square.
Advanced maximum likelihood methods like log-linear regression or Cox regression are helpful, and can be better alternatives to the above. However, unless you’re doing analyses every month, it may not be worth the effort to learn them.
It honestly depends on the type of work you are completing for your website.
You might be testing a new design out, just to see if your new design is usable. Or you could be completing A/B tests, which are tests that are conducted to compare two similar designs to see which is more successful. Another thing you might do is benchmark your existing website, to help expose pre-existing usability problems.
For usability testing, I would argue that you need to know very little statistical knowledge to understand and interpret usability measurements.
However, there are certain things you can and should quantify, such as:
Quantifying these three things should tell you if a user is capable of successfully completing a task with ease. Red flags (severe issues) are raised when they cannot complete a task, have many errors when completing a task, and/or take a long time to complete the task. These metrics help you understand if what was designed is indeed usable.
In addition to quantifying these, you should qualitatively analyze a test too. What I mean by this is take notes and try to understand why the user is struggling. This is supported when using the "think-a-loud protocol," which I advise looking into if you are not familiar with it. Essentially it helps you understand the user's thought processes as they are going through the test.
As for Google Analytics, I'm not well-versed in the tool to make an appropriate statement.
