As part of a pitch, business are determining the usability of two software products and have come up with a questionnaire to ask a set of end users.

They have asked me to feedback on the questionnaire and I said I will come up with 5-10 questions that variously test the usability of the two software systems.

The software systems are document search and markup applications.

My question is: what are the best questions to ask?

At the minute I'm seeking answers to how learnable the software is, how easy it is to perform common actions, how fast users can perform common actions etc


As I think of it, the questionnaire should be voiced "comparing the two systems, which one did you find the easiest to learn? Why?"


Here's a few examples of User Experience Surveys.


They are generic usability survey instruments and can be adapted to specific products. You can go through the individual questions and use the ones that you think are better suited for your product. PUEU and USE look like they would work well for your case.

As a surveying strategy I would use a comparative usability study. Timothy Rotolo, UX Architect at TryMyUI, has detailed why this would be a good approach when testing multiple products/designs:

'For comparative usability, it's typically better to use a different set of testers for each website or design rather than having the same testers try out both, for 3 reasons:

  • People naturally tend to prefer whichever one they use first, as it will set their expectations for the second one. Things that aren't necessarily bad about the second design might get a bad reaction because your testers were expecting it to be like the first one.

  • Having people use both designs can easily cause tester fatigue by running too long and burdening your testers with cognitive overload. Tester fatigue leads to unreliable, low-quality results, especially towards the end of the test.

  • Using two separate pools of testers for each design allows you to collect two totally independent sets of quantitative data for objective comparisons and measurement.'


Hope this helps. Cheers!

  • that's super useful – colmcq Apr 5 '19 at 12:12
  • These points about 1st preference and fatigue are kind of superficial. It's a classical statistical study problem concerning within- vs. between-subjects design (- NN Group), which is easily countered by randomization of the starting option, as long as the test is not too long overall. – Big_Chair Apr 5 '19 at 12:37

Firstly a general comment about study design:
As mentioned in a comment, both between-subject and within-subject design for studies are valid approaches for system comparisons. See NN Group explanation of these terms here, which basically says:

Summary: In user research, between-groups designs reduce learning effects; repeated-measures designs require fewer participants and minimize the random noise.

But apart from that you want to ask your own questions. Here it's important to use a system that has been proven to work (e.g. not make up your own rating mechanism without statistical proof of its validity).

Using a rating system based on the Likert Scale is a very common and valid approach. Quoting Wikipedia:

It is the most widely used approach to scaling responses in survey research, such that the term (or more accurately the Likert-type scale) is often used interchangeably with rating scale, although there are other types of rating scales.


example of a likert scale (image source)

The main point is basically to give the user 5 options to rate a statement, ranging from very negative to very positive. A neutral option in the middle is usually given too.
If more precision is needed, it can be 7 points. If a neutral response is undesired, it is also common practice to omit the middle option and only present positive & negative options.

Also, a common option to counteract "auto-pilot" ratings is alternating between positive and negative wordings, see more here.

An example question for your test could be:

Using the common action feature was effortless.

Now the user can rate how much he agrees with this statement.

  • one of the first things i learned in HCI was Likert scaling. – colmcq Apr 5 '19 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.