Should usability testing of a competitor's website be a part of your usability testing routine?

Question

I was reading up on best practices on usability testing for websites and a recommendation was that get your participants to do usability testing on your competitors website as well as that would help you better understand how they navigate sites like yours (e.g. an eCommerce site). Though I can see the merit in that, a lot of websites have different design patterns and user flows and I am not sure if there would be any real benefit in doing an usability test of the competitor website unless the site is very similar to your own site.

Has anyone done a similar exercise and has had results which show key takeaways even though the competitor site might not be a mirror image of your site. I would also love it if there are any references to published case studies or research articles.

Answer 1

I can think of a number of things this may do for your test results.

The obvious: consider the situation where your product isn't yet available, or you're developing a new feature. Learning from mistakes others already made, or from what they did right, is immensely valuable.

It may also help your judgement of the problems the test reveals. I think we all sometimes fall into the trap of downplaying the shortcomings of something we created, while we have no trouble being quite critical of the work of others. In testing your competitor's products you may find shortcomings you were overlooking in your own product. You'll look at it with fresh eyes and it can provide a reality check looking at your own product (again) can't. Especially if your doing an expert/heuristics review this could be very valuable.

Users will have learned certain problem solving tactics that they'll apply without being conscious about doing so. They might be very handy doing something on a competitor's site, but if that tactic is not available in yours it will not appear in your test. They might tell you something about your product isn't very handy, but they probably won't be able to tell you what. So, looking at people using your competitor's product might reveal such tactics.

Same goes for features in a competitor's product you may not have thought were important. Users may not tell you they use them, but tests might reveal that they're important anyway. Note that this means that the fact that your product and that of the competitor are not the same could be valuable too.

Something else: you're inviting people to judge your product. Of course you'll prepare them carefully, but they may be holding back criticism. Looking at a competitor's product could solicit feedback you were not getting on your own product.

Although this does not strictly relate to usability testing, I've found that in context mapping sessions anecdotes and comments that happen around the actual session can be very informative. Unless your testing for a specific problem area, this to me means that breaking away from the core test can give you a lot of useful input for the (interaction) design of the product.

There's probably more.

Answer 2

There is value in doing (usually quick and dirty) usability testing on some competitor's websites, but this should not be a UX focus.

Doing this is usually more about market research than UX, and is done mainly to understand where you have potential improve your product over theirs and where to focus your differentiation strategy.

That said, you should regularly be using other websites and products to help get ideas for how you can improve the UX of your own products and keep up to date with how other people are solving similar UX problems.

Answer 3

No, these are two completely different activities. Usability test of your site is one thing where looking at competitors site doesn't do you any good in that context.

However, you should have scheduled activities to benchmark your site against competitors every quarter to find out where they are now and possibly where they are going. This is in this case a part of a bigger marketing plan where UX is one factor. Pricing, terms and conditions, shipping and bundlings are other factors to measure in a benchmarking session.

Keep them separate but keep on doing both, and you're on top of things.

Answer 4

Yes. I've done usability tests on multiple competitor sites on several occasions for a few different reasons.

• As a way to judge the level relative ease of use on the various sites (using tools like SUS). If you are significantly better/worse than your competitors it can affect how you invest in future features.

• Learning about features in competitors sites (Does feature X look good? Do users like it? Does it seem to affect behaviour in good/bad ways?)

• To examine similar features (Does their checkout work better? Does their taxonomy make more sense to the user? etc.)

Answer 5

Most usability practitioners agree with the ISO/IEC definition of usability, that states that it's a measure.
It is a measure but there is no defined unit like meter or pound or jakobs to express it in absolute terms.
This lets you with comparisons only, like whatever its value, the usability here is more, or less, than there.
So if you can charter a group of subjects to try in the competitive site a very similar task as the one you are testing on your site, you'll get a glimpse of the relative usability values.

Answer 6

I have used others sites in user sessions a couple of times, although I would not call it usability testing as such. If you have the time and budget then this is a good user research activity. It is best done after the aims and scope of a project have been defined.

It's worth adding that the sites selected don't need to be competitors but need to be comparable in what they do. My aim has been to determine which sites do what they do well, what worked well for end users and how the existing site (if there is one) that we are redesigning compares.

The method was to use 10 sessions in which we picked 4-5 different sites with the clients site mixed in. After the initial warming up part of the session we would then go through a series of steps for each site with a number of possible routes to explore if a site drew up a dead end on one of them. For example when opening the site the users where asked to give their reactions to the site and then what do you think this site does and then if they where looking to do x where would they start.

Even in an hours section we could gather a lot of interesting information about what users where expecting and what they where not. This allowed us to create an informed set of 'best practice' recommendations as opposed to the 'expert' guides that I have often created.

One thing we learnt, for example, is that putting pictures of people on a website does not make the site warmer and more friendly, as is often said, but alienated people as they saw those people as not the kind of people they could relate to. People are very judgemental. Instead people preferred representations and close up shots of hands and tokens of wealth like pens and watches. In this case the client went on to redesign their printed material to take this into account.

So in summary - it's a great technique for quickly understanding what customers are looking for in a new site. You don't need to use competitors, just comparable sites and it's not a validation technique but a research technique and it is not the same as a customer audit - that happens prior to inform this activity.

Answer 7

Even if your competitor's website is slightly different, his approach can inspire you.

You don't learn exclusively from similarities, but from differences too.

The true similarity doesn't rely on formal aspects, but on a common principles. If there is a common principly with regard to your competitor's website and yours, there is always a possibility of adaptation, like in a mathematical function.

All you have to do is identify that common principle because of which a certain business is your competitor, than adapt his UX solutions to your case. It's all a matter of (+) and (-) which can be translated into your Use Case scenario.

Answer 8

I would suggest, and agree if already touched on, that the output of doing this would create bias toward any further work you conduct, to your detriment.

It's all fair and well reviewing competitors but you have to do so with specific intent, know what outputs you specifically want and how you are going to learn from them. Otherwise you run the risk of the that information adversely effecting what you do next.

Answer 9

Can be useful if your competitor has a very similar process to yours. Ease of use can make or break a transaction (users are very quick to compare), so if for some reason you think (or know) they have an advantage in the process, it could be a good idea to review the user experience, if not to copy, to try and present something better.

But, of course, this is more of an strategic approach than pure usability inspection.

Answer 10

I'd like to present a slightly different take on the question by asking the question "what do you qualify as a competitor?"

Reason I ask is that you could feasibly call either an existing service due to be re-designed a competitor, or at a push an earlier iteration of your own service.

With this view point, there is certainly value in understand with existing services what users like and don't like, what they find easy and hard, can complete or can't complete. This provides you insight into your users and their usage of existing services presenting your with opportunities to exploit, or mistakes to avoid.