"people don’t know what they want until you show it to them"

Question

One of the most successful products ever produced is the iPhone. It’s loved for its excellent User Experience worldwide. It came from an idea by the late Steve Jobs who is among other things famous for not asking the customer what they want. In 1985 he said:

We built [the Mac] for ourselves. We were the group of people who were going to judge whether it was great or not. We weren’t going to go out and do market research.

. . . In addition, in 1997 he said:

A lot of times, people don’t know what they want until you show it to them.

At the same time User Experience experts here on UX.SE and elsewhere say you need to test ideas, mockups, prototypes and do A/B-testing, open and closed Card Sorting and the like on the intended audience. Question is do we need to test, or don’t we?

Answer 1

These are very popular quotes from Steve Jobs and are frequently pointed to as reasons why we shouldn't test. If Steve did it, so can we!

The trouble is that most people miss several key points. The first is that they are not Apple:

A Forbes article, Five Dangerous Lessons to Learn from Steve Jobs, lists these quotes as #1. Chunka Mui, the author of the article, calls it out:

I actually think Jobs was right but only in the very narrow category to which he aspired: where his products, such as the Mac, iPod, iPhone and iPad, either redefined or created product categories. That’s not the domain in which most businesses play. Remember also that Jobs backed up his unique insights with an enormously expensive creative process populated by world-class designers. Without Jobs’ talents and the unparalleled creative team and processes that he built around himself, you won’t get away with doing no market research and not listening to your customers.

Another is to confuse statements about market research with user testing. Apple does do user testing!

In 2010, an iPhone 4 was found in a bar. Thing was, the iPhone 4 didn't exist yet - as far as the general populace was aware! It certainly wasn't left there on purpose, but instead likely by an employee who was testing it.

Another key point is with wording: want vs. need

People/users know what they "want". They "want" a lot stuff. They often times don't know what they "need"!

Steve Jobs understood this. In another article, The Steve Jobs Market Research Quote Should Rest in Peace, two different types of market research are discussed. It is clear that Apple does conduct market research, but they ask the right questions -- they look at what people need out of a product, not what they want out of it.

(Note: I'm not implying people "need" an iPhone, but what they "need" out of a product is very different then what they "want" out of it)

Broadly speaking, there are two types of research that work--and they are completely consistent with Jobs' words and success. Market Research that Jobs Agreed With First, you must spend time understanding and gaining insights into consumers' existing habits, beliefs, routines and unmet needs. We would never ask people, "What should we develop to make your life easier?" Instead, we spent time with people in their homes and watched them at the store, then we dug through data about what they are buying and using. Your job at the early stage of innovation is to get into the shoes of your customer, understand her life, and look for insights that give you ideas on products that you could create that would surprise and delight her. And this is exactly what Jobs was so good at. He saw early on that people were interested in using computers, but were frustrated at their complexity. He saw that people loved music, but couldn't stand loading a few dozen songs at a time onto the early MP3 players. Jobs deeply understood how people engaged with technology.

The site UXMyths hits on this as well: Myth #21: People can tell you what they want, which has a quote from Jakob Nielsen (among many other excellent example):

Jakob Nielsen says that the “critical failing of user interviews is that you’re asking people to either remember past use or speculate on future use of a system.”

To some extent, Myth #29: People are rational also hits on this point.

A last point, people always point to the iPod, the iPhone and the iMac as examples of why these quotes from Steve Jobs show "user testing doesn't matter." So, let's look at all the other awesomely successful products Apple has produced:

For some reason, people forget how often Apple (and Steve Jobs) has failed; and there are a lot more examples then just those above.

Question is do we need to test, or don’t we?

Yes. Yes we do... and we can use Apple as an example of why we should! For two main reasons:

1. Because Apple does test their products
2. Because sometimes when you introduce something so new it flies, but more often it fails.

UPDATE:

To extend on the "failure" concept, per a few comments below. The Newton was created sans-Steve, as was the Pippin. So, here are a few more products under Steve's watch to fill the gap of those two:

My point here is that Steve/Apple have failed plenty of times, some past and some more current-ish (G4 Cube). So to put any quotes by Steve as evidence to, or not to, do something because of the recent iPod/iTunes/iPhone success is ill-advised. They absolutely got it right, this time.

I point above that Steve/Apple does do testing, they do perform market research, and Steve also built a very talented creative team in Apple's later year that didn't exist in the beginning. But the "magic" doesn't always happen.

Answer 2

We're mixing two phases of the product development lifecycle.

Jobs' quotes were focused on Focus Groups--where you are essentially asking the consumers what they want. This happens at the beginning of the development lifecycle (or even before that in discovery). Often, if you rely heavily on focus groups to drive product design, you tend to end up with mediocre product. See also: Hollywood and Homer Simpson's car.

(Above: Homer Simpson's Car--My favorite analogy for what you get when you rely too heavily on focus groups)

A distillation of Jobs' philosophy is that consumers can only ask for what they know. And Jobs was really about pushing beyond that and giving the consumer something they never would have even thought of on their own.

Testing, on the other hand, is validating the product that you came up with. This happens during development of the product. Steve, himself, was known as perhaps one of the most ruthless and demanding product testers that ever existed. There are many notorious stories of Steve going into a fit over the smallest of details that he felt didn't meet his testing standards.

Answer 3

We need to test precisely because "people don't know what they want until they see it."

The usual way of addressing that quote is iterative design. Produce a partial prototype -- which may be a completely dummied-up set of interactions with hardwired inputs and outputs, in the early stages -- to act as a concrete illustration of what you intend to build. Let folks who know the customer/problem domain see it and tell you whether it addresses their needs or not. Repeat this process until you have some confidence that your solution (which may not be the one they were expecting -- and that's the other side of the quote) really does what they, and you, want it to do. Then expand that prototype, adding functionality, and continuing to test against domain experts on a regular basis.

The nice thing about minimal-function/incremental UI tests is that they're relatively cheap and easy to try if you get another idea; the nice thing about iterative design is that you're likely to discover you're going down a rabbit hole before you've sunk your entire development budget.

"Any sufficiently advanced technology is indistinguishable from a rigged demo." Used correctly, that's actually a useful development tool.

Answer 4

You're basically comparing a product and how to use it. While Jobs' (and Ford's) quotes are true, they refer to the product itself.

How am I supposed to imagine or a Mac (or a car), if I have never experienced one?

If we say "we need to test if this works", we're almost always referring on how to use the product. And this has to be tested.

Also, with market analysis and user testing, we try to minimize the possibility of market failure.

Let's be honest - not every Idea we have is a Mac or a car.

Answer 5

I think it's all different facets of the project Process.

In terms of conceptualizing and market research, that is something that happens before creation therefore market research is not reliable.

A/B Testing and prototyping etc is for after the ideation is through and you are working on refinements and assessing market acceptance.

Process should look like this:

1. Figuring out the needs.
2. Ideating solution.
3. Go out on a limb and run with it to produce Product Prototype.
4. Market testing.

Answer 6

I really believe users don't know what they need and what they want is not necessarily what they need.

But that doesn't mean you should just forego any user testing. Sure you can do without, but you user testing can give you insights you (and your team) might never have come up with.

When creating a product you can withhold from doing market research, because you firmly believe in the need for the product. As for the first Mac, Steve Jobs and his team had a need for it. That's a small market research in itself and it was all they needed. Their success is due to the fact (or the luck) the need exceeded further than just to their team.

I don't believe they didn't do any testing after creating the first Mac (or other product). Actually, I believe they did so many user testing they finally found out through trial and error that they couldn't just ask the people what they needed and that they got the best results when they just observed the user and construct needs from the users' behavior.

So those quotes from Steve Jobs shouldn't lead you to believe user testing is obsolete, but it should teach you how to perform user tests correctly.

Answer 7

Let me take quite a bit of a different approach to this question than the other answers.

For good UX and UI design it is necessary to know what the user expects and figure out how they use their current products and design accordingly to make the product good. For this you can do market research, various forms of testing, etc.

For great or (r)evolutionary UX and UI the trick is figuring out better designs. Minimizing the number of clicks, movement, etc. and teaching the user new ways of interacting with content is what you do for this. You can still do testing, but due to the learning curve that often comes with this approach tests are unreliable and to do them reliably you need a far bigger budget.

The disadvantage of the second in the end is that there is a far bigger risk factor (see Evil Closet Monkey answer for that), whereas - as pointed out by L. Möller - the first approach can minimize the possibility of market failure. As en exemplification of this you can look at answers here on UX stackoverflow and some argue based on what implementation's currently do

Site x, y and z do not make the description clickable

and from there (optionally) reason why that is the case, whereas other answers will start with

Making the area clickable will make user interactions overall more efficient as less dead clicks will happen

and (optionally) show cases where this is done that way.

Now, with certain things you can do this quite easily (like making the entire description clickable), but let's say that all dialogs would have the cancel button at the far bottom right and you would be the first to figure out that having the 'ok' button in that place is better - which we just assume to be the case here - then doing this randomly will still confuse users as it against (and not only different from) what they are used to. So in conclusion there is always a cost to innovation and you either need to be really big or really new to pull of these kind of things.

Either way, if your intention is to shake up the market and you think you have a long breath and are willing to take the risk then testing comes secondary and figuring out the best design comes first. Now, let me just finish of with a few cases:

• Facebook: Facebook is reiterating it's design continuously and making quite bold decisions from time to time. Nearly every time there is a huge backslash from the community, yet despite that Facebook is still doing it. Why? Because not doing it would cause them to be outdated after which everyone would switch to a small new competitor which can pull of the change.
• Windows 8 Metro/Modern design: Although from a UI standpoint I am not a fan of it, the same applies here. One of the problems however is that it seems they did after all do a lot of short term testing and due to this made some half hearted compromises here and there.
• Or take Google Search, as far as testing goes they are doing an incredible job. They roll out experiments and keep them running for months and they actually take the time to see the effects. It's not a luxury most companies have, and even the more exotic changes are rolled out in small increments over long periods, so they are (overly) careful, but they are still able to make quite big changes in the end without negative backslashes.

Over all, do test no matter what, just decide whether you want to take risk and make the design first and then test, or play safe and test first and then design (and test again).

Answer 8

I would just add that "people don’t know what they want until you show it to them" is not at odds with testing. Coming up with a new idea is often a solitary process, and not just a matter of asking people what they want. But then you do have to show it to people and tested before you know if that idea is actually a good design or successful product.

The mistake is to think that all new features should be based on customer requests. Sometimes they will, but sometimes you should take creative new idea and test it.

Answer 9

One have to separate how to design something and how well the design works. A visionary can see a need - even if it's not invented yet. Nevertheless - the design should (when in such shape) be tested.

The important thing is: evaluate how the user uses and interacts with the product, don't ask the user how it should have been done.

Answer 10

Focus groups to determine what people want are simply a waste of time. That is what Jobs and others have found pointed out.

In other words people are often so caught up in their own world that they generally can't see beyond the problems they are faced with in order to tell you what would really help. So using a focus group to come up with ideas for a new product won't work.

Focus groups covering screen mock ups are of limited utility. Often times people don't really know what they'd like to see until they are already heavily involved in using the product. The type of feedback you generally get will be "move this button there", "I need a link here" or "I think this element should be pink" ... until they actually start using it. At which point they'll have an altogether different idea of how it should be laid out and look.

For incremental changes to existing systems, listen to the feedback. What are people struggling with? Is giving what they ask for the right way to solve the problem or is there something more fundamental at work? Can you solve two problems at once, perhaps one they weren't even aware of? Beware of this though as too much change can be either a great thing or the worst thing to do.

The iPhone was a completely different way of using our phones. Steve, et al, saw what people were struggling with - VM was a horrible interface, most phones made you jump through hoops to sync contact lists, texting was difficult and email was what you used the laptop for. Of course, the most important item was that NONE of the large vendors were spending the resources necessary to solve the problem. (Yes, I'm aware of Nokia, RIM and others were "working" on at the time.) This meant that market ripe for a remake; which Apple happily did. Visual voice mail, a decent email client and actual syncing with your desktop contacts being the TOP reasons for it's success.

Have Apple, and others, attempted to solve problems people just didn't have? Yes. Have Apple and others solved problems but in a underwhelming way? Yes. "Network Computers" such as the Pip pin being one idea that industry watchers at the time thought would take off but which people paying attention knew wouldn't - we had already moved away from dumb terminals why go back?

Evil Closet Monkey listed a number of Apple failures; and yes, as individual products each of them were failures. One thing I'd like to point out though is that each failure was an iteration on a theme that ultimately led them to the iPhone and later iPad. The Newton was trying to solve having a worthwhile computing device in your pocket. The one button mouse was trying to simplify computing. Ditto for the mac mini. When those ideas were applied to telephony and combined with new technological improvements (ie: relatively cheap touchscreens and faster, smaller, less power hungry processors) then you have a winner.