User Experience Stack Exchange is a question and answer site for user experience researchers and experts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

It seems like in a business-driven organization (i.e. where the sales and marketing team has a bigger say) that offer software products or services, A/B testing is used as a way to improve and measure 'UX'. I have looked at many of the websites that offer this service and more often than not they simply reveal practices that seem contrary to good user interface design practices, and therefore changes to the design appear to provide the boost in conversion rate. On the other hand, there is a school of thought that failing quickly so you can make changes seem to make sense in an 'AGILE' software development environment.

I believe that A/B and multi-variate testing as it stands covers only a small part of the UX design, and that there is too much weight given to the conversion process. However, it seems that many companies prefer to do this type of evaluation compared to a proper study of users and a user-centric approach to product development.

What are the parts of UX design that can be incorporated into A/B and multi-variate testing to cover not just the business related goals, but also the customer/user experience improvements?

share|improve this question
This is sort of like asking if you can replace a car with 4 wheels. The wheels are a part of the car, not a replacement for one. ;) – DA01 Mar 27 '13 at 5:12

First of all, design evaluation and user testing are parts of UX design. You cannot replace the whole with a part (atleast not here). The problem here is, both A/B and multivariate testing are quantitative measures. You are not at all covering the qualitative measures of the users. Their emotions, their expectations, and other matrices are not visible if you just keep on performing A/B or multi variate testing. You can include a survey or feedback at the end of the test to get some of the qualitative information rather than just number crunching the results of the tests. The users always have some form of expectation from the system, most of the time they will be able to tell you/point you in the exact direction they want you to implement.

share|improve this answer
Depending on how the test is conducted, I think you can get both qualitative and quantitative measures on an A/B test as well. (e.g. what colours is more effective at getting attention from the user?) – Michael Lai Mar 27 '13 at 3:22
That is still quantitative information ;) You're getting a % or some other value representing how 1 color is better than the other, you're getting no explanation on why it is so. – rk. Mar 27 '13 at 3:29
I guess I am talking about measuring qualitative (e.g. colours) or quantitative variables (e.g. number of links), not just the results that you get back. In a sense the quantitative information you get from A/B testing is still qualitative because it is a probability value that you have to decide the significance (with some interpretation). – Michael Lai Mar 27 '13 at 3:33
Yes, but the interpretation is missing in that result, which is the point I am trying to make. Unless you interact with the user and get their feedback, the results tell only half the story. – rk. Mar 27 '13 at 3:36

The downside of A/B testing is that you know that something happened, but you don't know why it happened. You can try to gather this information via quantitative methods like surveys, but users aren't necessarily able to articulate the reason why they did something.

A/B testing is only one tool available in the toolkit of user research. There are many ways to use it effectively. There's the saying that when you have a hammer, everything looks like a nail. For those who only know of A/B testing, it seems that every question can be answered via A/B testing. It's the job of the user experience team (or lone UX professional) to identify the right question to answer, and the right method to answer that question in a way that is actionable by the team in the time that they have available.

share|improve this answer

A/B testing can help discover barriers to completion of a goal. But user experience is more than driving behavior. It's about designing for how a user feels about the product they engage with from their interactions. Our thoughts and decisions, our experiences come from emotions not from rational constructs. A/B is one tactic in a strategy for designing that experience.

So to answer your question, apply the A/B testing tactic to the emotional aspects of product interaction. Examples:

  • A thank you email sent on a purchase vs none or two different messages. (generic or from Sandra)
  • "We think you will like this."
  • Member since date

Imagine A/B testing on the presence or variations of these things to see if they induce desired or unexpected behaviors without specifically guiding a user down a path to completion.

I hope this makes sense.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.