I'm working on redesigning an old e-commerce website and after running a few qualitative user research studies, I have three designs were identified to be AB tested. To give some background on the AB testing, there is web metrics data that gives information on what the users do, but not why they do it.

The basic data seems inconclusive, so I am not sure how select a design. Is there a best way to go about this?

  • That's a really broad question. Normally you'd look at the web data in terms of user flows, and work out if there was any stage in the user flow where a high percentage of users dropped out, then you'd take a closer look at those places with qualitative research.
    – Racheet
    Jan 22, 2014 at 11:03
  • 1
    You cannot get more information out of the data than what is contained in it. AB testing is not supposed to deliver data which tell you why the user does something; you need to choose other study methods if this is what you want to know. And as for how to decide, well, if your current data is inconclusive, then there is no way to make it conclusive. If both designs are equally good at the criteria you measured, either find a new criterion and measure it too, or pick one design at random.
    – Rumi P.
    Jan 22, 2014 at 11:33
  • @Racheet. Thanks that's helpful. The management wants us to be a little conservative with changes. Though qualitative research has been done to the entire flow. The changes to the actual website is one page at time. For the A/B test: We’re tracking the user engagement (clicks to different parts of the page) and the monetary value of the designs options.
    – Tara
    Jan 22, 2014 at 19:28
  • 1
    Results: engagement x, a, b are almost the same range with a variation of +/- 2%. And the engagement value for (x, z) is about -3% but the hypothesis is this variation provides lot more information other variations. Monetary value for each variation is about +/- 3%. You've a good point about flow. I'll look at that as well. Also, what kind of qualitative research are you suggesting?
    – Tara
    Jan 22, 2014 at 19:30
  • @Tara that depends on your budget, anything from hallway testing up to a full eye-tracking study. The point is to find out why users are unable/unwilling to complete the task at the place where the data says they are dropping out.
    – Racheet
    Jan 23, 2014 at 12:00

1 Answer 1


Are you comparing an old design to a new one? If so, and the variance is so little, stick to the old design. This is because you haven't fully tested the variances over a long period of time, and therefore if your new design is not obviously better, then you're risking a long-term dropoff for no measured gains:

Tyler Roehmholdt, Web Marketing Manager at Campaign Monitor

For an inconclusive test, oftentimes we’ll stick to the control if it’s a change to something that already exists on the page. If we went with the new variation, at best it will perform the same as the original over a longer period of time, at worst we’d see a decrease in conversions over time. If it’s a new addition to the page, we’ll be much more open to promoting the variation to production.

In fact, researching this, there's a good Stack answer here.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.