User Experience Stack Exchange is a question and answer site for user experience researchers and experts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

As I mentioned in this question, I've been working on proof-of-concept applications for social network privacy and running usability studies to evaluate them for publication. I'm a beginner to user studies at best, and I'm ashamed to admit that I don't think I've been asking the right questions. I've generally asked users to compare the simplicity and usefulness of the applications to the privacy controls provided by other social networks, such as Facebook, but I'm curious if there are even basic questions I should be asking.

My questions are:

  • Are there any basic questions a user should be answering in any usability study?
  • Specifically in a comparative usability study, are there additional base questions a user should answer?
  • Can a user usefully "quantify" the usefulness or simplicity of something, even in a comparative case, or should questions be of a binary nature?
  • In a study trying to quantify the usability of an application versus simply make improvements, is there any advantage to asking open-ended questions?
share|improve this question
Have you examined the standard comparative scales like the System Usability Scale? – Ben Brocka Apr 16 '12 at 15:36
up vote 5 down vote accepted

Before writing a survey, first do open-ended qualitative interviews, even if they're informal and brief. Here's a slideshow with tips about how to do that:

It'll give you a much better sense of how people think about the tools that you'll be surveying them about... things like the terminology you use, the way you phrase the questions, etc.

Quantifying questions aren't uncommon, but they're usually phrased more the way people think about them. For example, "I do x... Always /Sometimes / Never" vs. "What percentage of the time do you do x, y, and z?"

share|improve this answer
+1 for interview users (to understand them) before writing formal questions. – PhillipW Apr 16 '12 at 9:50

I strongly agree with Kimberly on interviewing user up-front for better context regarding what they will do, which will help you understand why they're doing it.

An important distinction - Do not blend market research (MR) and usability. Your interviews with the participant prior to the test should be to get a better understanding of their predisposition to the functionality you'll be testing. The test should be about their ability to complete a task.

You can supplement this with things like "time on task", "completion rates" or "user's ease of use rating" if you're trying to benchmark.

You should, however, avoid preferential questions or questions which assess a value to the product or feature. These questions should be left to market research studies.

In short - usability is less about whether the business should move forward, tweak or position a product in a certain way. It's about how to convert a known business problem (proven via MR) into a usable product.

p.s. To quantify use a Likert scale for responses, as Kimberly mentioned.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.