How can this discrepancy be reconciled?
You have divergent results because the number of participants is small and not representative. There is no randomization or blinding to prevent bias. You're also not calculating the relevant stats. (What are the standard deviation, margin of error, confidence intervals, odds ratios, p values, etc?)
Further, you ...
How to reconcile the discrepancy? That I can't tell but here is why it might have happened.
The "5 users will find everything wrong with your system" refers to usability problems test subjects will find in your usability tests. Sauro has a great article that goes quite deep in to this "5 users is enough" thingie.
The five user number comes from the ...
I just looked at http://isitjustme.de/2012/01/crowdsourcing-the-translation-of-sus/ users user13154 recommendation. It seems that the effort was picked up by SAP and there is a professionally translated and verified version at https://experience.sap.com/skillup/system-usability-scale-jetzt-auch-auf-deutsch/
Ich denke, dass ich das System gerne häufig ...
Look at your confidence intervals: the "real" score for v1 is somewhere between 58 and 88, while that score for v2 is somewhere between 51 and 79. Most notably, the confidence interval for each score contains the mean for the other score.
What this is telling you is that your sample size is too small. Based on the data you've collected so far, you cannot ...
Jim Lewis wrote an article in 2012 for Measuring Usability
The article explains the direct correlation found between SUS scores and Net Promoter Scores (NPS). They found that it was possible to predict NPS scores based on SUS scores, which means the inverse is also true.
The Net Promoter Score is a widely used survey which consists of only one question. ...
A success rate is one of many metrics used for measuring/quantifying usability. As http://www.measuringusability.com/blog/essential-metrics.php describes, if a task can not be completed, the product is not usable. "If users cannot accomplish their goals, not much else matters."
The methods used for determine success rate will vary greatly depending upon:
For me it's a swings and roundabouts discussion. Each has some advantages and disadvantages.
Here are some pointers to some research / background that should help you decide between them:
A comparison of current approaches to usability measurement.
SUS: A Retrospective
A Comparison of Questionnaires for Assessing Website Usability
(and if you go for SUS I ...
A quick google search lead me to MeasuringU:
The average SUS Score is a 68: When looking at scores from 500 products, we found the average SUS score to be a 68. It's important to remember that SUS Scores are not percentages. Even though the SUS ranges from 0 to 100, these scaled scores are not a percentage. A score of a 68 is 68% of the maximum score, but ...
This small study suggests that the quality of translation is important, especially for question 8:
There is a crowdsourced German translation which has improved the German translation, which confirmed that it is reliable (and I believe SAP now use it):
Measuring Usability has an article that briefly touches on sample size. TL;DR: 2 is the obvious minimum, 5 is a suitable starting point.
From 10 Things To Know About The System Usability Scale (SUS):
You can use SUS on small sample sizes: One common question I get when using the SUS (or when measuring usability in general) is about the lowest acceptable ...
Concerning the second question, Blažica & Lewis detail their method of translating SUS to Slovenian in their paper A Slovene Translation of the System Usability Scale: The SUS-SI. They use method of back-translating and psychometric evaluation to validate the translation.
There were three stages in the translation process. First, 10 reviewers from the ...
As with any metrics or scores for the online/digital space, you have to fully understand exactly what you are comparing to make sure that you are making meaningful comparisons.
This why you normally wouldn't try to compare between two different types of rating systems, because they are designed for different purposes. Also, it is only meaningful to compare ...
Recent research has brought into question whether or not it is OK to break the SUS into Learnable and Usable subscales -- for the full article, see http://uxpajournal.org/revisit-factor-structure-system-usability-scale/ -- here's the abstract:
"In 2009, we published a paper in which we showed how three independent sources of data indicated that, rather ...
When analyzing survey data you need to separate validity from reliability.
Validity is a measurement of how correct the answer is.
Reliability is a measure of how consistent the answer is.
Here's a graphical representation of the concept (from Wikipedia):
In general, SUS has been found to be highly reliable and there is some strong support that it ...
You need to set your business goals and based on that you can set you ways to measure the success.
Think about an e-commerce site, or think about a social network. The goals of the business will be completely different, i.e. % abandoned baskets, conversion, etc.
Tools like Google analytics help to make sure you are able to constantly monitor the site and ...
In Germany these two questionaires are used (and I think known in usability circles):
I'll cite from my answer here: https://ux.stackexchange.com/a/72905/3311 :
The AttrakDiffTM questionnaire by Hassenzahl et al. (2003), developed
together with User Interface Design GmbH, measures subjective
assessments concerning ...
You can find the version that we use here: http://minds.coremedia.com/2013/09/18/sus-scale-an-improved-german-translation-questionnaire/
In the development of the German version of the scale, we analyzed and included the other sources mentioned in earlier posts and eliminated a problem with one item.
The scale was used with > 80 participants, without any ...
System Usability Scale questions
I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be
able to use this system.
I found the various functions in this system were well integrated.
I thought ...
You can have several levels of "success rate", but in essence it's a matter of saying "yes" or "no" to this question:
"Did the user accomplich the task?"
Quote from the book, p65:
To measure task success, each task [...] must have a clear end-state.
It's similar to "effectiveness" in the ISO 9241-11 definition...
The ten questions of SUS are not "quite a lot to ask" if you only administer the SUS questionnaire once at the end of all the tasks (which is how it's meant to be used). The thing with the SUS is that although there are 10 questions, they all fit the same basic format ("here is a statement, and a 5 point scale of strongly-agree to strongly-disagree"), and ...
As others have mentioned, picking a few questions at random from SUS isn't going to give you something that you can usefully use as a guideline.
If you're interested in specific factors then I'd take a look at Lewis & Sauro's paper on The Factor Structure of the System Usability Scale where they've done a bunch of research around that topic. That may ...
Regarding the question, "In particular...is there anything which produces somewhat similarly reliable results with fewer questions?"
You should take a look at the published research on the Usability Metric for User Experience, published by Finstad (2010). The UMUX has four items and has typically been found to have desirable psychometric properties and to ...
Both printed and digital documents can be evaluated for usability, in the sense that you can define metrics that gauge just how easy it is to use the document. There are also overlaps between the use of colour/contrast, typography and information architecture that are relevant, although you have to adjust it for the type of content, which is a static ...
What pairs of questions test respondent's consistency?
They don't exist ;-)
The SUS wasn't designed with pairs of questions in mind. The SUS wasn't designed to test particular factors (although there has been some interesting post-hoc factor analysis of the SUS).
To quote from the original SUS paper:
SUS is a Likert scale. It is often assumed that a ...
Generally speaking no, they are not. The global UX described by the SUS may vary from the general UX described by the WAMMI. So you are basically trying to compare different things that have a similar 0-100 scale.
Because SUS works very well and has been examined closely practitioners for more than 25 years.
The SUS questions weren't pulled out of a hat. They were research based. To somewhat extensively quote from Brooke's original paper:
SUS is a Likert scale. It is often assumed that a Likert scale is
simply one based on forced-choice questions, where a ...
The SUS is given to people who have used your product in some way. Your method of having them perform their task and then administer the SUS is a perfectly valid application of the SUS. It's very flexible.
I think the most important thing is to realize that using a combination of qualitative and quantitative analysis methods is going to give you the most complete best picture of what the users are thinking and doing rather than relying on one or the other (which as you can see led to different conclusions). The idea is to use quantitative analysis when you ...
With SUS, you have some room to play around with the words without effecting the results too much.
For example, you could change it to:
I think that I would like to use this service frequently.
I think that I would like to use this product frequently.
Whatever works in your context.
This is backed up by this article.
Small changes are less ...