0

We are interested in running a purely quantitative remote web-based user test with 100 specific users. Where we show them different design variations of a concept and ask them to vote for the one they like best. Then we will tell them the purpose of the design and ask them to vote again. This test would be sent out to targeted users, conducted remotely and self-service via a questionnaire/presentation type format.

The goal is to get another level of feedback about some new design variations we are trying in addition to the more qualitative user testing sessions we are doing.

Are there any key things or best practices we should be aware of and/or focused on when trying such an approach?

Improve this question

4 Answers 4

Reset to default
1

Voting on what someone likes best is not helpful. It does not provide a reason for liking a design.

What if I vote for a design because it uses my favorite color? How is that useful? You wouldn't even know why I voted for it, and might assume it was because of an illustration.

You could select the winning design and then in iteration unknowingly change what people liked about it, because you wouldn't know the determining factor.

If you want to know if a design is effective, you could frame the question such as this:

In the above design, what do you think happens if the user clicks on the star icon?

A. Goes to astronomy section

B. Views latest celebrity news

C. Displays top-rated lolcats

This way you can get specific actionable information rather than just what random people like.

Improve this answer
1
  • 1
    I'd add to Eric's answer, too, to make sure you randomize the order in which users see the options you'd like feedback on, just so you don't accidentally induce a bias as a result of user fatigue or anything. Too, I'd reiterate strongly that you don't ask for people's 'preferences'. What they indicate doesn't at all tell you what works - infer the best design from a series of in-context questions. Maybe run a separate test for each design variation, keepung the questions the same for each, and see which performs best on its own? Different set of users each time.
    – Sam
    Oct 19, 2016 at 21:16
0

For quantitative data, Try converting some qualitative questions into quantitative ones in the following way.

For. Ex:

Does the visual design look aesthetically pleasing? Can be replaced with

On a scale of 0-10, 0 being the very bad looking and 10 being the best looking site that you have seen, how much would you rate this website.

Using this approach, you can get several quantitative data but not limited to just good or bad

Improve this answer
0

General Rule: The more time / 'thinking' required of users - the less likely they are to complete testing.

If you are asking them to do something which requires more time and effort, you might need to incentivise them more.

Improve this answer
0

If you are doing comparison study for the designs (comps) here is a list of parameters suggested for the quantitative analysis. If it is an Interactive high fidelity prototype, there are lots of other parameters we need to check.

  • Likert scale
  • Affordance test
  • Branding Test
  • Comments / Feedback from the participants
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.