How to user test a redesign?

Question

In some website projects, you are testing whether users can complete specific functionality. For example, with a shopping cart I can measure the percentage of users that complete checkout, the amount of time it took them to complete checkout, etc.

However with something like redesigning a user's dashboard, it seems like metrics might not be the best indicator of whether to go ahead with a redesign. You can test things like, "can people find the billing page", but that doesn't take into account the aesthetic.

What sort of questions can you ask during a user test to evaluate whether a new site design is better than the old site design? If you are just going on "how it looks", then the HIPPO (highest paid person's opinion) will win every time.

Answer 1

Just because a matter is subjective doesn't mean it's impossible to measure in a fair user test. Subjective measures exist and you can easily craft your own survey using simple Likert Scale questions.

Craft a subjective measure regarding your site's visual design and let your users tell you how good each design is. Ideally you should use a Between Subjects design, however a Within Subjects design can also work if you stagger the experimental conditions.

Sample questions might be:

• This site is well put together
• The appearance of this site matches its tone
• I trust this site
• I enjoy using this site
• The images and graphics of this site work well
• This site is ugly

Logically group questions and compare group or single question scores between the two designs. If the average score for design 1 is higher than design 2 for questions like "This site looks good" then design 1 is probably more aesthetically pleasing.

Another resource you could use is Usaura, which allows you to take static screen shots of your design and have people perform one of many tests. You can have users click on the image they prefer more or answer multiple choice surveys. An advantage of Usaura is that you could gather your users online instead of in person.

Answer 2

If you're trying to measure the 'softer' qualities of the design (aesthetics, emotions, etc.) take a look at Attrakdiff, a questionnaire focusing on the so-called hedonic qualities of the design, based on some solid research. I've used it before to quantify differences between two versions of a program.

Answer 3

so to avoid the lock in effect to affecting the evaluation of your new design by older users, show the new design to new users only. Then show old design to the same new users. In other words, do an A/B testing with new users only. This is the only way to cleanly know if a new design or redesign is good. See also the QWERTY lock in and path dependece Now you know which design is better but that does not mean you won the hearts and minds of the older users.