Isn't all user testing essentially a way of gathering how people think things should work?

Question

When user testing is performed, users are often presented with a serious of variations of a product and observed to see where they excelled and where they had trouble. People will often click in areas that are not intended to be clicked on, or may try to achieve a goal in a way that you had not designed for. Now put yourself in their seat for a moment. When you go to click on something, why are you doing it? You are doing it because either the interface has made it clear what to do, or you are simply relying on how you think things should work.

Once the results are gathered and analyzed and conclusions are made, the research is published and becomes factual information that others, like ourselves on this site, use to help create their products and interfaces.

So step back for a moment:

Isn't all user testing essentially a way of gathering how users think something should work and then making conclusions from that research? (seeing as all decisions that are made while being tested are based on how the user thinks the interface works

If so, then how are what ones thinks of the best way to do something (like on this site) often only taken with a grain of salt when they aren't backed up by user testing/research? Yes it is very true that using research to support your claim strengthens your argument and essentially says "I believe this and 100+ people in this study agree with me". But how is this any different that 100+ people agreeing with your individual opinion that is not backed by research?

Answer 1

You are conflating 'subjective' and 'unreliable'. Usability tests aim to get reliable information about people's reactions. Self-reported opinions are also subjective, and are much less reliable indicators of how other people will react to the interface.

If I test 100 people and their subjective opinion is that they hate an interface, I'm pretty sure that the next 100 people I test are also going to hate it. That's reliable data. And it's because usability tests are structured to factor out a lot of the unreliability.

If I ask 100 people, "Do you hate this interface?", the data will be all over the map:

• Some folks will try out the interface and answer to the best of their abilities. But they may feel they have to suggest or shade their feedback.
• Some will want to please me and answer yes or no accordingly.
• Some will feel like they did a good job (and say so), when actual evidence would reveal they are less productive, made more mistakes, or otherwise found the interface less than ideal.
• Some will hate something peripheral to the interface. (Think of all the 1-star reviews of appliances on Amazon that are really 1-star reviews of UPS or Fedex.)
• Some won't even use it, but will make up an opinion anyway. Then they'll retroactively come up with reasons--that may seem very real to them!
• People who feel strongly are much more likely to respond. The "silent majority", however, make up the bulk of your users.

Quite frankly, most people are very bad at self-reporting accurate impressions of an experience.

Usability testing sets users up for success:

• We exert control over what they're experiencing
• We ask them to explain their expectations and actions
• We have the opportunity to guide them down specific paths
• We have the opportunity to ask follow-up questions
• We can compare very specific portions of the interface to understand why a user has one opinion or another.

In short: usability testing is a process whereby professionals can mitigate biases in subjective experience to understand the impact of a design.

Answer 2

This is a really important question. Not all research is created equally. Honestly some published studies are fundamentally silly, but you have to evaluate each on its own merits, which is why the "methodology" section of any scientific paper is so important. It matters how results are obtained and analyzed and we would all do well to be more constructively critical of UX "studies."

Generically though, the reason I give more weight to studies (if well crafted) than self-reported or opinion data is that people lie. Take, for example, the question of how much someone drinks or if they go to church. When self reporting, respondents invariably drink less than consumption data would indicate and go to church more than attendance stats would indicate. Self reporting is far more subject to personal bias than is measured behavior. And that difference is interesting too, but if I want to design products/interfaces based on what people actually do and succeed at doing, rather than what they wish they did, I'll take the measured success data any day.

So that users do better or worse at some assigned tasks in studies, even if it's because of their cultural or educational biases or training is still an important finding--though it's application may be limited in time or to a specific audience. Back to the methodology, it matters who's in the study group and their demo and psychographics. The generalizability of the findings are constrained by the characteristics of the participants. But that's important and why I would still design a slightly different interface for senior citizens than I would for college students even for the same content/domain.

As another example, I've also done continuous customer satisfaction research for many websites and after a site redesign, we almost always saw self-reported satisfaction levels drop. So, in a nutshell, return users thought the redesign sucked. However, concurrent with that same data, we would often also see conversion rise and/or inbound call center calls drop. Eventually (a couple of months) the customer sat levels woudl recover and rise above their previous levels. Bottom line? Users aren't good designers. Who cares what they think they think? It only matters what they do and that's not always the same as what they say they think.

Answer 3

The user test is the research.

Ideal user testing scenarios will collect both quantitative and qualitative data. For the quantitative, you typically will want to test something very specific, and usually as compared to something else (a/b) testing. Those situations produce a result typically "x% accomplished the test with option a than with option b".

Qualitative is where you can collect ideas/suggestions from the users that aren't as easily measured into digestable data points. It's more about exploring other ideas/options.

Now, some people only deal with qualitative (like Hollywood Test Screenings) and some ignore qualitative research altogether (Steve Jobs) but most tend to straddle the fence between the two as much as they can.

Answer 4

There are plenty of sorts of testing that aren't especially subjective. Think of studies that define

• The average completion time for two forms, differentiated by only one UI strategy
• The mean eye-tracking heatmap for a statistically significant sample size
• The rate of 'correct' (sane) responses to forms with particular UI elements

Yes, you are measuring human opinions and subjective responses. But one can collect objective data about subjective things. It may be statistically invalid if it has a poor sample size, but that is not the same business as something being 'subjective'.

Answer 5

User testing allows to detect inconsistencies between the mental model your target audience creates about the system and the model of the system defined by the designers.

To detect inconsistencies, individuals which represent the target audience are evaluated and a generalization is performed to discriminate individual from general inconsistencies.

Is this subjective?

The mental model each user creates is influenced by many different elements (conventions, previous experiences, cultural, etc.) which are most of them subjective. The key is that our target audience will be also influenced by these elements and will have similar problems with the system.

There are exceptions that do not apply to the target audience but this happens with any statistic process.

Why not to just ask for the user opinions?

The user's inability to predict their future behavior is widely documented. So, the strategy followed is to detect the problems instead of asking for the solutions to the user.

Answer 6

Yes, but its not the whole picture. When the user acts towards our interface with a goal in mind there are two influences. Their own opinion and then the interface itself. The interface and experience can influence their behavior and even their opinions on how it should work.

So, User Testing is gathering peoples opinions and perceptions as a metric to influence and be influenced by a UX design.

Answer 7

Just answering part of the question:

A basic rule in psychology is 'What you see - is what you think you'll see'.

One's entire experience of the world is 'constructed' based on existing 'models' in the brain.

The brain is pretty good at this, but things such as visual illusions, demonstrate where the brain can't get a mental model to match the incoming data.

Most of this processing is, however, unconscious. So the conscious 'user' is a bit like an iceberg: they only have access to 1/10ths (or whatever) of the processing which is going on in their brain. So they know what they are seeing. But they generally can't tell you why.

Answer 8

The difference is the background of the opinion. If you have a bunch of individuals each posting their individual opinions, there is no way to say that one is more valuable than another. All you can do is personally agree or disagree with them.

However, if you have a person stating "this is what a usability study with 100 participants shows" then that is going to hold significantly more weight. Yes, it is still just opinions, but those opinions have been collected and analyzed and consolidated to a formal conclusion.

Think of it like a political poll. You saying that your local mayor (or governor, or any other elected figure) is doing a horrible job and should be removed from office is really not that valuable. But a poll of a decent-sized sample of voters where the majority are saying the same thing definitely is valuable. Those opinions by themselves really don't mean much, but as a whole can provide a barometer of how the official is doing. Usability testing is the same way.

Answer 9

Perceptions and opinions are not the same thing. Perceptions are our understanding of the world based on what we have learned through our senses. Opinions are our value-laden thoughts about something (e.g. about something we have perceived). How something is perceived by an individual is a subjective fact ("I can't see the button because it is the same colour as the background"). If the evidence/stats stack up and you can be confident that just about everyone will have the same perception then you have a normative fact (users will not see the button). If the button in our example really can't be discriminated from the background because it is the same colour or due to limitations of human physiology then we have an objective fact.

Users will also provide their opinion ("This interface is great, I like how it makes me hunt for the buttons"). Their opinions matter in a number of ways. They will tell others what they think of your interface, and the perception of your interface will be influenced by the opinion given.

Usability testing is about gathering normative facts. It can also be about gathering facts about people's opinions about the subject of the study (the software). Both can be valuable. They are however different things. Research that takes people's opinions (about the software) and reports them as facts (about the software)—e.g. instead of saying "45 out of 47 users said that the software was difficult to use" the report says "45 out of 47 users had difficulty using the software"—is bad research. It is flawed. It should not pass peer review. You should not give it any credibility.

There are varied reasons why facts-about-opinions and facts-about-behaviour-and-performance with software shouldn't be confused and misreported, but one important one is that they don't always match. People can report that interface A is horrible and slow and they made many more mistakes than they did with interface B etc etc where this is their opinion but the evidence shows that actually they performed better and made fewer mistakes with interface B. And the reverse can also be true. This isn't to say opinions don't matter. But facts about opinions and facts about behaviour/performance/etc shouldn't be confused.

Answer 10

During interaction between an end-user and system ( Web and Mobile interface, Kiosks, etc), user physiology and psychology work hand-in-hand to develop overall human experience framework.

1. What a user does online, e.g., Objective data-->action data, Performance and web analytic)
2. What a user feels and thinks, e.g., Subjective data-->emotions, satisfaction, expectations, engagement and attention)
3. Physiological data: How physiological impulses drive human psychology and overall subjective and objective behavioral process or emotions. ( Eye-tracking, EEG and NIRS Brain signals, GSR and Body sensors)
4. User Demographics and context of use attribute work as moderators of inferential relationship.