Should the design/appearance of an application a part of a usability test?

Question

So i was wondering, i developed an application (android) and put alot of effort in creating a user friendly and visual appealing experience for the user.

Afterwords i held a small usability test with a few users. The test was pretty simple. The test consisted of a few tasks which the user had to solve (use cases), a questionary part after each scenario and a survey to address general issues afterwards.

In the general part i asked: "How would you rate the design/appearance of the application?" (The users had to give a rating from 1 (bad) to (5) good).

At the time i thought its a good question and the answers really helped me to find a few errors in my design. But now i have to write it down (master thesis) and i'm struggling, because I got the feeling that the answers do not hold any quantifiable meaning. Because design is only subjective to each indivual. Which leads to my question:

Is it bad practice to ask users, how they would rate the design/appearance of any software prototype? If yes, how would you guys handle this subject? Would it have been better to ask specifically what the application is doing right or wrong, visually wise? Or am I just stressing out and I simply could use the information i gathered?

Answer 1

Yes indeed

While it's correct taste is subjective, it's also true that taste is a cultural construction in constant change, and there many specialties around this discipline. Visual design is one of them, and you can use countless factors to measure its effectiveness, both qualitative and quantitative.

Take a look to the sample below:

I don't need to do any study to know more than 90% (if not 100%) of users will choose the option above. I can measure color impact, usability, readability, subjective taste, time spent on page, eye tracking, you name it. So, while the concept of taste as as subjective concept is true, it's also true we can even measure subjectivity by the way of related actions and interactivity . The tools to measure this will vary depending on the kind of data you look for. You can use tools like Likert or Semantic Differential scales, or measure time spent or track gestures... well, options are really broad, not to mention you can research by the way of semantics, semiotics, iconography, symbology and many many many more aspects

Additional Resources:

• Visualizing Research Strongly recommended!
• Design Research: Methods and Perspectives
• Visual Research: An Introduction to Research Methodologies in Graphic Design
• The SAGE Handbook Visual Research Methods (24 pages extract from the book, short but really useful)