How do I know why people prefer our competitor's web site?

Question

First let me explain the context: We have a web site that helps people to find jobs (e.g. https://www.careerbuilder.com/ https://www.glassdoor.com/index.htm )

In our country, there are some web sites that have the same purpose but there is one prefered by people. Obviouly it isn’t ours.

We want to do a research about the perception of our site and our competitor in order to find out why people prefer the competitor.

I thought about making some kind of A/B moderate live testing: Recruit several users, half will test our site and the other half our competitor’s web site. The same tasks will be performed by both.

But I sill don’t know if this will work.

Could somebody recommend me a test that help us to obtain significant feedback? Or is there another way to find out why people prefer our competitor’s site?

Thank you very much.

Answer 1

There could be lots of reasons why people prefer the competitor, ranging from habit, lack of awareness, or legit usability issues. "A/B testing" both sites would compare apples to oranges, because you aren't reviewing your customer's full journey, nor are you keeping business goals and user goals in mind. You'll just be comparing solutions to a problem you don't know yet.

You should probably chart your customer journey and your users first if you haven't already. Information gathering can be done through interviews, surveys or shadowing. This will clue you in on user goals, motivations and needs. You need to know all this before you can truly start designing or address any potential issues.

Preferences are very personal. Data can give you an estimate reason, but you won't truly know unless you ask.

Answer 2

I think a better use of your time would be to interview the users as opposed to testing them, but before you even do that I would define who your users are by mental models. You need to distinguish between where users that you are targeting are in the journey.

If they are fresh and never heard of you before, they may not prefer the other website, they just don't know about you, so maybe it's your marketing and nothing to do with the product you made or the competitor's product. As some stated above, it's not necessarily a UX problem. So you're messaging would need to adapt for this type of user to even sign up. So then you would go test that messaging.

If they are an existing user, and you know that they switched to a competitor then that's the person you really want to talk to as to why that it is. What you need to find out is why they choose to stay at the competitor and either replicate what the competitor does or do it better than them and incentivize these people to come back over.

Also many or some users may be shared between both of your sites, so "more" is a relative term if the same users sign up for both and are just using a "shotgun" approach to finding a job. They may not actually prefer one or the other. So this is like your "indifferent" archetype. Defining your users first into thinking modes like this will help you construct more targeted testing later. You can always improve your product but you should know whom you are improving it for and why before aggressively testing.

Answer 3

Interesting question! I have a couple of suggestions that have worked for me in the past.

1. Identify the core user journey for your site, that your competitors also offer. This is what you're going to test against.
2. Create two user tests. Test 1, starting with competitor core user journey vs your user journey. Test 2, starting with your site core user journey vs your competitor's user journey.

You run two tests and flip the competitor and your site to try and alleviate bias to which site is easier/harder to use etc.

To find accurate participants for your test, you can do a couple of things:

1. Use your user's for the test. You could potentially run the risk of pointing out flaws in your site compared to your competitor + there is the potential for bias toward your site.
2. Use a recruitment service and find participants by matching a screener question, like currently looking for a job, hasn't used site X etc. There are agencies that can do this for you (costly and timely) OR you could use a remote user testing service that usually allows this recruitment.

I hope this help. Feel free to comment back if you have any questions or need help :)