3

Can SUS be used in order to test the usability of a document?

Any guidelines will be much appreciated.

  • Printed or Digital document? – Benny Skogberg Jul 13 '14 at 4:38
  • The digital document. – Navi Jul 13 '14 at 8:04
  • E.g. e-bill document. – Navi Jul 13 '14 at 18:46
2

Both printed and digital documents can be evaluated for usability, in the sense that you can define metrics that gauge just how easy it is to use the document. There are also overlaps between the use of colour/contrast, typography and information architecture that are relevant, although you have to adjust it for the type of content, which is a static document rather than an interactive webpage.

To provide some guidelines, it would be good to know the type of document and the content involved, but here are some areas to look into further:

  • Presentation of content, which includes colour, typography and layout
  • Organization of information, which includes structuring and breaking up of content into logical flow
  • Ease of reading, which includes readability tests and also related to the first two areas

You can probably try to adapt the SUS to ask these types of questions if you want, but it might be easier also to just construct questions around these topics.

1

I wouldn't use it.

I'd use a test which was designed to measure "Readability"

There are a number of established tests of this

http://en.wikipedia.org/wiki/Readability

This covers the content of a document but not other aspects such as font, font size, and indexing.

  • And what if users need to do some operation (calculation, analysis) with the document, would ASQ (after-scenario questionnaire) be more appropriate? – Navi Jul 13 '14 at 22:52
1

SUS has been shown to work well with isolated parts of the system - i.e. you can re-define the "system" being surveyed as "the eBill".

However if the phrasing of a question causes significant doubt about a valid answer, then that is likely to skew the results. So take look at the questions individually

  1. I think that I would like to use this system frequently. (ok)
  2. I found the system unnecessarily complex. (ok)
  3. I thought the system was easy to use. (term use won't cause significant issue)
  4. I think that I would need the support of a technical person to be able to use this system. (just ok - 'technical person' will be interpreted as 'someone to explain the bill')
  5. I found the various functions in this system were well integrated. (what functions? if no interaction then some significant doubt here)
  6. I thought there was too much inconsistency in this system. (ok)
  7. I would imagine that most people would learn to use this system very quickly. (ok)
  8. I found the system very cumbersome to use. (ok)
  9. I felt very confident using the system. (ok)
  10. I needed to learn a lot of things before I could get going with this system.(ok)

Given the above, if the eBill has interactivity I would run the standard SUS test with "eBill" given as the "the system" under review.

However if eBill is static then Q.5. would need a re-visit. e.g. "I found the various parts of the eBill were well integrated." Which could introduce a small variance to the scores. I would expect that most people would understand that a tweak that small will not impact results.

Likewise I would replace the text "system" with "eBill" throughout the questionnaire rather than just defining it upfront.

I'd recommend validating these tweaks with any reviewers before doing survey.

  • @ Jayfang Thank you very much for your answer. That exactly what I had in mind. I thought changing system to eBill like you advised and for Q.5: is it OK to ask something like "I found the varous items of the eBill were well integrated?". My intention was to ask participant to do some sort of bill analysis and calculation and then give them this usability questionnarie. Based on this, would it be OK to ask e.g. Q.1 "I think that I would like to use eBill frequently to verify my expenses?" or something in that direction? – Navi Jul 14 '14 at 9:46
  • Think about it from the users terminology. If "items" is likely to be interpreted as "invoice line items" then it will be a mis-leading term w.r.t. intentions of SUS survey. SUS is implicitly saying 'does the information architecture and workflow make sense to you'. A bill is organised into 'sections' 'parts' or ? – Jason A. Jul 14 '14 at 9:53
  • Well yes, there are actually different part of bill each containing different information about the name of services used, amount, price, basic info, etc... – Navi Jul 14 '14 at 10:02
  • The example of revisited Q.1 seems correct to you? – Navi Jul 14 '14 at 10:35
  • Good question. Q.1 frequently seems an ill fit where usage is infrequent or periodic by it's nature - and there are many such scenarios. So in this regard Q.1 is not a worse fit in this case than for many others. Thus the SUS results will not be skewed. Also credit most users with common sense. They will interpret this as 'will I be happy to see this eBill every time I am due a bill' and answer accordingly. – Jason A. Jul 14 '14 at 12:11
1

@Jayfang :

  1. I think that I would like to use eBill frequently to verify my expenses.
  2. I found this eBill unnecessarily complex.
  3. I thought this eBill was easy to use.
  4. I think that I would need professional support to use this eBill in order to verify my expenses (analyze).
  5. I found the various parts in this eBill were well integrated.
  6. I thought there was too much inconsistency in this eBill.
  7. I would imagine that most people would learn to use and analyze this eBill very quickly.
  8. I found this eBill very awkward to use and analyze.
  9. I felt very confident analyzing this eBill.
  10. I needed to learn a lot of things before I could get going with this eBill and do some analysis.
  • In general I would not go with that degree of modification - because even if "better" it introduces variability away from standardised scores. Especially any goal qualification such as "... to verify my expenses" would be a red flag. SUS should have open goals - whatever the user want's from that part of the system is what matters - not what you think they would be doing. – Jason A. Jul 15 '14 at 9:11
  • @Jayfang Would it be the same drawback of saying "easy to understand" instead of "easy to use", or "able to understand" instead of "able to use"? – Navi Jul 15 '14 at 9:34
  • Just "easy to understand" is likely to have small/no impact. It depends on the sensibilities of those that will be reviewing the results. – Jason A. Jul 15 '14 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.