A/B testing - how to deal with minority that chose B?

Question

Publishers can test different site layouts and various versions of their content (for example, testing more than one headline on an article)

This can be done with A/B testing. For example, you might find that 60 percent of users prefer layout A, and 40 percent prefer layout B. You go with A because that majority of users preferred it.

But what about all those people who preferred layout B?

Answer 1

'C'.

Option C should be a working model that introduces the compromised hybrid solution that attempts to find a way to appease both testing groups to a larger percentage. It'll always be changing, refining its processes through many micro updates that gradually change the UX. This, of course, requires a ton of resources, your undivided attention, and so much more -- so often times the answer is 'nothing'. You don't do anything about the people who prefer option 'B'. You found a way to tip the scale, and settled with layout 'A', because hey, you can't please everybody, right?

Answer 2

As with most UX activities, there should be an overall objective and the process involved should align to meeting the objective.

A/B testing is a common technique used to help guide design decisions, but it should not be the only input considered, so the first thing is to understand what the results mean in the context of the overall design decision to be made.

Often the results provide clues to doing more in-depth research to uncover user needs and requirements, so it can be the first step rather than the last step in making design decisions.

In this case, a general answer to your question would be to:

• Consider how the result fit into your existing knowledge about users
• Decide whether the result is significant enough to warrant further research
• Uncover the underlying user needs and translate them into design decisions
• If it is a trade-off (i.e. A or B), then work out what the alternate use case is and whether it can be designed to cater for both groups.
• Implement the changes and test to validate the assumptions

So you can see that you might not actually end up with having to 'deal with' the minority that chose b, since there can be designs made to cater for both cases or it may turn out that the results are not validated when the changes are implemented.