Conversion based on clicks or orders

Question

I'm running A/B split testing using Instapage.

My dilemma is whether using clicks or orders versus visitors should be my conversion rate.

This is what I had so far

A: 556 visitors - 27 conversions (clicks) - 4.85% conversion B: 543 visitors - 15 conversions (clicks) - 2.76% conversion

The Significance test (http://getdatadriven.com/ab-significance-test) shows that the testing was significant.

Obviously not all clicks on the "buy now" button convert to an order. The other thing is what if 100% people ordered from B but only 5% from A? By looking at the clicks I don't know.

Of course if I look at my statistics I can guess how many orders did each variant generate.

My CTR before testing was 2.09% and my conversion rate (based on orders) was 0.55%. With a bit math I can then "assume" the following:

A: 556 visitors - 6 conversions (clicks) - 1.07% conversion B: 543 visitors - 3 conversions (clicks) - 0.55% conversion

The significance test however indicates that test was not significant, even though the ratio of conversions are about the same (A converting 50% better than B).

So using clicks seems like scientifically more significant if the number of orders are small.

Should I redo the test based on order as conversion metric? Shall I use order metrics for future tests?

Answer 1

You need to have a hypothesis for every test. With that, you will have your answer.

Is it:

Changing XYZ will increase clicks significantly

or is it

Changing XYZ will increase revenue significantly.

When you start adjusting your success metrics based on what gives you better results, you're contradicting the benefits of running statistical experiments in the first place.

Another way to analyze it is:

If revenue (or orders) increase but engagement drops a lot, will we want to use this as a winner?

vs.

If clicks and engagement increases tremendously, but revenue and orders decrease, will we want to use this as our winner?

I would bet that almost always your orders and revenue are way more important than clicks. But this will also depend where you are testing, and how much traffic you have to support that deep of a test.