1

I'm running A/B split testing using Instapage.

My dilemma is whether using clicks or orders versus visitors should be my conversion rate.

This is what I had so far

A: 556 visitors - 27 conversions (clicks) - 4.85% conversion B: 543 visitors - 15 conversions (clicks) - 2.76% conversion

The Significance test (http://getdatadriven.com/ab-significance-test) shows that the testing was significant.

Obviously not all clicks on the "buy now" button convert to an order. The other thing is what if 100% people ordered from B but only 5% from A? By looking at the clicks I don't know.

Of course if I look at my statistics I can guess how many orders did each variant generate.

My CTR before testing was 2.09% and my conversion rate (based on orders) was 0.55%. With a bit math I can then "assume" the following:

A: 556 visitors - 6 conversions (clicks) - 1.07% conversion B: 543 visitors - 3 conversions (clicks) - 0.55% conversion

The significance test however indicates that test was not significant, even though the ratio of conversions are about the same (A converting 50% better than B).

So using clicks seems like scientifically more significant if the number of orders are small.

Should I redo the test based on order as conversion metric? Shall I use order metrics for future tests?

  • think it's more practical to be looking at how many visitors made it all the way thru your checkout flow as well. – Ameen Akbar Mar 19 '15 at 10:50
  • So would you redo the test? – Barnabas Nagy Mar 19 '15 at 15:00
1

You need to have a hypothesis for every test. With that, you will have your answer.

Is it:

Changing XYZ will increase clicks significantly

or is it

Changing XYZ will increase revenue significantly.

When you start adjusting your success metrics based on what gives you better results, you're contradicting the benefits of running statistical experiments in the first place.

Another way to analyze it is:

If revenue (or orders) increase but engagement drops a lot, will we want to use this as a winner?

vs.

If clicks and engagement increases tremendously, but revenue and orders decrease, will we want to use this as our winner?

I would bet that almost always your orders and revenue are way more important than clicks. But this will also depend where you are testing, and how much traffic you have to support that deep of a test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.