Given data:
30 datapoints in the form of usability scores (e.g. SUS) 2 groups within data
Objective:
Investigate statistical significance of difference between usability scores of two groups
Problem:
Basic statistical instinct says that for non-normally distributed data independent t-test is not appropriate. Instead, non-parametric testing e.g. Mann-Whitney U should be used. However, stumbling across following resource stating:
While the normal distribution is the reference distribution used in most of the statistical procedures we recommend, it is the distribution of the sample mean which needs to be normally distributed.
Question:
Now, what is the appropriate approach here to investigate statistical significance comparing two independent groups?
Preliminary approach based on research:
For larger sample sizes, the underlying distribution of data becomes less relevant as indeed the distribution of means again becomes normally distributed. Therefore, for larger sample sizes, tests assuming a normal distribution such as t test are robust and acceptable (See central limit theorem). Often large enough is considered to be 30 per group.
However, for smaller sample sizes, it is better to apply tests which do not make an assumption about undelying distribution. Still, if the underlying data is normally distributed, it is more likely that the test then rejects the null hypothesis and identifies no significant difference.
