User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
My client is running a series of vendor demonstration where they want the staff to measure the perceived usability of the system (SUS and SEQ).
Originally we asked for a consistent group of evaluators but alas due to a "scheduling malfunction" this isn't possible.
Any suggestions on how i can mitigate the risk?
On the contrary, all research emphasized how important it is to use the actual user group in your test and not the contractor or leadership.
Despite this, I believe that your planned studies are better than nothing. Every usability specialist has some kind of "quick and dirty testing"-advice or "do it yourself"-advice. And these advices do usually include some randomly picked person from the street.
You just need to be aware of this bias when you analyze your results.
To validate your findings and to compare the result against the real user group, you could pick a small sample and perform a correlation analysis.
If you could get some data on your actual user group, you can calculate the correlation in eg. Excel:
Col B and C are the average findings from the SUS. Col D and E are the comparable measures (ie. for question 1,3,5,7,9: D=B-1 and E=C-1, and for question 2,4,6,8,10 D=5-B and E=5-C). The correlation formula in C14 is probably "=CORRELATION(D2:D11;E2:E11)" (I have the Norwegian MS Office edition)
A high correlation indicates that you probably would get the same result in both groups.
Building on Jørn's answer: statistical analyses can help you compare a small group of high-quality users with the rest. Which tests are most appropriate will depend on your sample size and on the asssumptions you can safely make about the nature of your data (such as normally distributed, equal variances, etc.).
To compare the scores of two groups on the SUS, I'd start with the total scores (on the 0-100 scale). Check if the scores are approximately normally distributed (either visually in the histogram, or with a statistic test such as Kolmogorov-Smirnov or Shapiro-Wilk), then calculate the mean and standard deviations for both groups. Your null hypothesis is that scores are the same for both groups and that the differences in scores between the two groups can be attributed to chance. If you can assume equal variances, you can just use a independent sample t-test with unequal sample sizes, otherwise use the variant for unequal variances Welch's t-test. You can do this with any spreadsheet or statistics program, or use one of the online calculators such as this one from graphpad.
Sample calculation: group A (10 real users): mean 68, standard deviation 17, n=10 group B (40 random people): mean 76, standard devation 19, n=40
Unpaired t-test gives a p-value of 0.23 (not significant), so the difference between the means can be attributed to chance.
In case your results show that there is no difference between the two groups, I'd argue it's safe to assume that your data gives an accurate representation of what the real users would say and you can combine the two groups for further analysis.
If not, try massaging your data a little. The SUS is known to measure both learnability (in items 4 and 8) usability (in the rest of the items) (according to research). So try leaving out items 4 and 8 and run your tests for the usability dimension only.
All of what I've written above is for when you have no additional data besides the test scores. In case you're able to collect demographics (even basic information on session number could be valuable) you have a lot more possibilities.
