0

This might seem subjective, but lately i'm hearing the word benchmarking more often in the UX space. A few years ago competitor analysis was used more often. i'm curious are they the same thing or are they different or is one a subset of the other.

1
  • 1
    Terminologies can change over time or take on new meaning, so this is a rather broad question unless you can provide some context around where you heard the term used previously and now. I suspect that you won't get a definitive answer, but you should focus the question on the answer that you are looking for.
    – Michael Lai
    Jul 14, 2020 at 2:27

2 Answers 2

3

Short answer: It is what we make it.

Long answer: The UX community has a long history of confusing and subjective terminology. Often, the driving force seems to be "correct and right" rather than doing what we're trying to do for our users: provide value.

I have worked with die hard pros who has been doing UX before computers were invented - and with newcomers fresh out of school. Everyone so far in my, and their, career has been using a wide range of terminology interchangeably. No is right or wrong. Ever. It is just words. :)

What strikes me is that every time terms like these are used, no one means the same things. Ergo, I would say the most important for us to recognize what needs to be done and then do it. Give it a name if you want to, as long as the perceived value outcome of your method is your driving force.

Personally, I have used "benchmark" as a general term for whatever kind of benchmark we are doing, while competitor analysis can be a part of that, or something completely different. It has varied time over time.

I have come to a point where I've realised that "internal" vs. "external" benchmarking/analysis is pointless to separate. Our users never compare our product to just the previous version of it. We always compete with whatever previous experience and current expectations the user have. Thus, I always try to benchmark whatever I'm doing against "external" experiences, if that is even possible to separate.

Before ending this, I'll just mention one of the things that has been my biggest revelation during my work: your competition is very rarely the other company doing the similar thing you're doing. As an example, I worked with a travel service (flights) and when the team talked about competitor analysis/benchmark, they strived to be better than the current "competition". What is so hard to realise is that our little bubble within the current industry is not the reality of the user. Our competition was not another flight company, it was everything the user could pick between. Sure, we can compete with better UX onboard the plane or a better booking flow. But those users are easy to hook and win, they have already the prerequisites for taking a flight.

This needed to be redefined. When we did our competitor analysis/benchmark, we couldn't compete in the same little bubble with already flight-convinced customers. We realised, our competitors, and the lion share of users worth way more, is all the things making users not taking the flight at all.

How do we compete against the train? The ferry? The user's current memories of each mode of transport? Against fear of flights? Against not wanting to travel at all? Against current travel plans that do not match with the concept of flights? Against taking the car, or walking, or skipping it all because everything will be to expensive or time consuming?

Thus, we had to make a lot better. As the example with our booking flow, it had to be better not only compared to other flight providers, but against every other possible booking flow that could be involved in the service industry (ferries, trains, car rentals, hotels, restaurants, etc.). And the same thing about other key aspects.

Comparing and working against clearly defined competitors is hard. Doing it against abstract things like the user's previous memories and similar, is extremely hard.

The more I have adopted this mentality, the more I have actually skipped the hair splitting about methodology. :)

1
  • 1
    +1 some very good insights based on your past experience, and I hope that it is a view that more of us can also adopt as well (i.e. forget about the terminology and process and focus on the best way to solve the problem).
    – Michael Lai
    Jul 14, 2020 at 2:08
1

There is definitely an overlap in the use of these terms, although I would say that there is no definitive answer as to whether they mean the same thing or not (like a lot of terminology being used in this field).

If we take the broad meaning of the term 'competitor analysis', it can be referred to as any type of business, technology or user experience analysis that you carry out on your competitors. Therefore there isn't really any methodology defined other than the fact that unlike comparative analysis you are not looking at what isn't in your current market. The things that are in scope for a competitor analysis are also not defined but rather determined by the problem you want to solve (or answer you want to obtain). The value of a tailored competitor analysis is that you can focus on the information that you want to compare against other products or services in the same market, and is generally valuable when you are at the early stages of design.

So in terms of the output for a competitor analysis, it is usually presented as a list of properties or characteristics identified across a number of related products and services. This generally helps to highlight the presence or absence of features or qualities across the types of services and products available in a given market. It allows you to get an idea of what types of features are common, and whether your product or service might have a unique selling point that other products or services don't have.

Benchmarking is a common term that has a more specific meaning because it is intended to be a way to compare between things against a set of guidelines or standards. Thus there is a general methodology described but again the way this is done exactly and the scope will vary depending on the problem you want to solve. The value of a benchmark is that it allows you to compare your own products and services against a recognized or agreed set of criteria (e.g. measures of metrics that are shared). In addition, the more products and services that are benchmarked, the greater the value of the benchmarking guideline or standard. However, this information is often not available or is done by a third-party company that monetizes this information (e.g. Gartner's Magic Quadrant).

So in terms of the output for a benchmarking activity, you would expect firstly the guideline or standard to be defined clearly, as the score that you achieve is only meaningful in the proper context (i.e. the guideline or standard you use to score it). The benchmarking can be performed on your product or service, or any number of existing products or services for which no benchmark score is available for.

Something to keep in mind both for doing competitor analysis or benchmarking is that the results only capture a snapshot in time, and therefore it shouldn't be something that you continue to refer back to if what you need are current information. Because benchmarking is done against a guideline or standard, if those guidelines or standards change or are updated then you'll have to adjust your interpretation of results accordingly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.