Measuring the user experience of Augmented reality application for elderly

Question

I'm conducting a study on elderly experience of AR application. The plan was to use SUS (System Usability Scale) to measure the ease of use. Then, after planning the whole thing, I realized I made a stupid yet large mistake. My audience are Arabic Elderly and there is no translation of the SUS to Arabic.

My questions are:

1. What method might I use instead of SUS that will give more quantifiable results? I found other method but the mainly focus on website not mobile application or not suitable for elderly.

2. If I would use SUS, what should I do to validate the Arabic version?

I'm open to any suggestion.

Answer 1

Concerning the second question, Blažica & Lewis detail their method of translating SUS to Slovenian in their paper A Slovene Translation of the System Usability Scale: The SUS-SI. They use method of back-translating and psychometric evaluation to validate the translation.

There were three stages in the translation process. First, 10 reviewers from the fields of computer and natural sciences individually reviewed a draft translation. Second, the final translation incorporated their comments. The third stage was to perform a back-translation. Three independent translators, without reference to the original, translated the final draft back into English. The translators were native Slovene speakers fluent in English. For all 10 items, all three translators provided back-translations with the same meaning as the original and, in some cases, exactly the same wording. For example, Item 9, “I felt very confident using the system,” was back-translated to “I was very self-reliant when using the system,” “I felt very confident using this system,” and “I felt confident when using the system.”

Translated SUS was then used to test Gmail with 182 subjects who also provided the likelihood-to-recommend (LTR) score to be used with validation. Results were then analysed as followed:

• Reliability: assessed using coefficient alpha
• Concurrent validity: correlation between SUS and LTR
• Construct validity: two-factor solution of the SUS
• Sensitivity
• Normative comparison: comparing results to evaluation of Gmail using English language SUS.

For the first question I recommend the book Quantifying the User Experience by Lewis & Sauro, in which they present and compare different post-study and post-task standard questionnaires. They don't address the issue of this question but other post-study questionnaires to consider instead of SUS are UMUX with four items and UMUX-LITE with two items.

If you don't have to use post-study questionnaire, SEQ (Simple Ease Question) is a recommended post-task questionnaire and quite easy to translate. It just asks how difficult the user thought the task was on seven step scale and it could be asked by the person conducting the user test.

If you can't get your hands on the book, I'd recommend Sauro's website MeasuringU, if you haven't already checked it out.

Answer 2

As 'locationunknown' has pointed out, SUS has a very odd factorial structure: two of its items are actually about Learnability! Yep, read them and guess which! This means that SUS has an effective range of values between 8 and 40 and most people score in the upper ends anyway.

The SUMI questionnaire (sumi.uxp.ie) is a more reliable brute that I created and have known and loved for many years :)# HOWEVER it does not as yet have an Arabic translation. If you're still interested, contact me and we can have one up and running in about three weeks - once I get my head around how to write HTML from right to left (just kidding).

Because of the poor factor structure of SUS nobody bothers to validate it psychometrically - they just adapt it and blaze on. I have seen more variants of SUS than... [metaphor deleted].