Is it effective to ask users which design they prefer and why?

Question

I have a few different options for the UX of a tool. Each is quite different from the other.

They are currently represented as lo-fi wireframes. I would like to get feedback on them to decide which one(s) to move forward with without building prototypes of each.

Given I have access to a wide range of users would it be effective to simply show the users the different design options, explain how they work and then ask them which they think would work best (and why)? How would the quality of the results compare to a usability test on a prototype?

Answer 1

In short, no. People without the literacy needed to evaluate a design by looking at wireframes can't give you a good answer. The only way to get a real answer from your target audience is to test the design with them. Otherwise it's like a doctor asking a patient which treatment they prefer, rather than saying 'this is what you need.'

Testing doesn't have to be a big and expensive deal, either. You can go as simple as wireframing the screens and dialogs needed for a given task and asking them to 'tap' through them by pointing to screen elements and telling you what they are thinking as they do. Make notes about what they get right, and what wasn't clear to them, and revise. Read up on paper prototyping for some tips on how to go about this.

To go one step better, get the design prototyped so that your test audience can use it on a screen, and see how it goes. It's more time consuming, especially if you don't code yourself, but it's very effective.

You can also be quite limited with your test audience; 3-5 people will find the majority of problems, based on research by Jakob Neilsen. Even if you can only get one person who will be an actual user of the product, it will help you much more than asking them what they prefer. Testing is part of going the distance to producing good designs, and asking for preference seems almost likely to produce the opposite of a successful design.

Answer 2

You're designing a Tool (a Web Application) which means User Interaction can only be evaluated through....interaction. The degree of value that you will obtain from your user's looking at a low-fi wireframe will fall short of being valuable for any prototyping. In essence, you're asking for User Experience feedback on something that cannot be interacted with.

Your discussion with users should come before your shortlist of options and after the first phase of prototyping.

Answer 3

As Bill Buxton argues, showing different design options will help the user expressing critical thoughts: they can do so without risking to "throw over all your work".

So you could give some example tasks to "accomplish" (How would you try to do it?) on every design option, and then let them compare which they liked best, and even more importantly, why.

Answer 4

Users are notoriously bad at predicating what they may do in the future which is why I believe it’s a mistake to show users multiple designs and ask them which they think they would prefer to use. According to Daniel Gilbert, author of Stumbling on Happiness, “One of the most insidious things about side-by-side comparison is that it leads us to pay attention to any attribute that distinguishes the possibilities we are comparing”. In other words, “side-by-side comparisons will cause users to consider all the attributes on which the designs differ, and end up considering attributes that they don’t really care about but just so happen to distinguish one design from the other”.

In almost all cases it’s better to narrow down design alternatives to a single design, usability test and iterate.