User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
If I want to A/B test a button color, but for 5 colors, how do I do it?
Do I equally split traffic to all 5 button colors for 5 weeks
Or… do I run split traffic between red button vs blue button the first week, then after finding that red works better, run the red button vs yellow button the following week, and so on?
Both options seem like they would work, but I'm not sure which to choose.
You should test all colors within the same time frame. If you test sequentially, you reintroduce the "conflating variable of time", as Sam Blake explains in an answer to A/B Testing vs Cohort Analysis (emphasis added):
The point of doing A/B testing rather than cohort is that it eliminates the conflating variable of time. The data you gather is only valid if there aren't ulterior explanations for why two groups behave differently, and groups doing things at different times will often behave differently.
This appears to be an XY Problem. You are asking about the best way to test colors (Y) using A/B testing (X). However, A/B testing may not be the most appropriate tool to test color selection. So "All at Once" and "In Succession" would both be "wrong" choices.
If color is not expected to have any effect on UX, there is no point testing it. For example, the buttons on this SE site are blue. On another I use, gray. On yet another, red. It doesn't affect my use of the sites in any way. You are burning away resources that could be used to test something more substantial.
A/B testing is called A/B testing because there are usually two choices, A and B. While you can increase the number, it's not ideal. Resource usage increases quadraticly. Suppose 100 users are needed to test two colors. To adequately test 5 colors, you'd need 1000 users. Is it necessary to test five colors? Can the number be reduced? How much traffic do you expect to have? Will you have enough users to test that many colors?
I do not see any advantage on testing two colors against each other over testing all colors in the same timeframe. I would actually prefer testing all colors during the same timeslot. This gives you certainty that no other circumstances influenced your testing.
I think before doing A/B testing it is important to clarify the objective or hypothesis of the test so you can understand how to interpret the results and therefore what changes you should make based on your findings.
You might find that the effect of changing the button colour to help improve conversion rates (which is usually the case for call-to-action buttons) is not necessarily due to the effectiveness of a specific colour (e.g. blue versus red), but rather the properties of that colour (e.g. bright versus dark).
The two different processes that you have described has the potential to result in two completely different conclusions depending on what the optimal colour might be. For example, if red happens to be the most effective colour by a large margin, then regardless of the way you run the test you would probably expect the same result.
On the other hand, if red is only slightly better than all the rest of the colour but blue is much better than yellow, then you might for one week happen to see more clicks for blue and so you pick blue to compare with yellow next, coming to the conclusion that blue is the best colour. You might not be able to conclude that red is the best colour with a side-by-side comparison, but you can probably eliminate some colours that definitely should not be considered.
Consider the case that red buttons are better for one type of transaction or user flow, and another colour is preferred for another type of transaction or user flow. How can you determine the best overall colour to choose (since you will probably not use both or change them depending on context). In this case you may find yourself comparing red with blue for one type of transaction and then also for the other type of transaction.
I guess the point I am trying to make is that you should try to set your A/B tests to be as simple and definitive as possible, because the more variables you introduce (even if it is just more colours) the more likely it is that some other factors will complicate the interpretation of the results. It is hard to control all the factors to make sure that the only factor that has an impact on the user behaviour is the one that you are testing, and to attribute that effect to the variable you are introducing.
