Building on Jørn's answer: statistical analyses can help you compare a small group of high-quality users with the rest. Which tests are most appropriate will depend on your sample size and on the asssumptions you can safely make about the nature of your data (such as normally distributed, equal variances, etc.).
To compare the scores of two groups on the SUS, I'd start with the total scores (on the 0-100 scale). Check if the scores are approximately normally distributed (either visually in the histogram, or with a statistic test such as Kolmogorov-Smirnov or Shapiro-Wilk), then calculate the mean and standard deviations for both groups. Your null hypothesis is that scores are the same for both groups and that the differences in scores between the two groups can be attributed to chance. If you can assume equal variances, you can just use a independent sample t-test with unequal sample sizes, otherwise use the variant for unequal variances Welch's t-test. You can do this with any spreadsheet or statistics program, or use one of the online calculators such as this one from graphpad.
group A (10 real users): mean 68, standard deviation 17, n=10
group B (40 random people): mean 76, standard devation 19, n=40
Unpaired t-test gives a p-value of 0.23 (not significant), so the difference between the means can be attributed to chance.
In case your results show that there is no difference between the two groups, I'd argue it's safe to assume that your data gives an accurate representation of what the real users would say and you can combine the two groups for further analysis.
If not, try massaging your data a little. The SUS is known to measure both learnability (in items 4 and 8) usability (in the rest of the items) (according to research). So try leaving out items 4 and 8 and run your tests for the usability dimension only.
All of what I've written above is for when you have no additional data besides the test scores. In case you're able to collect demographics (even basic information on session number could be valuable) you have a lot more possibilities.