I'm running A/B split testing using Instapage.
My dilemma is whether using clicks or orders versus visitors should be my conversion rate.
This is what I had so far
A: 556 visitors - 27 conversions (clicks) - 4.85% conversion B: 543 visitors - 15 conversions (clicks) - 2.76% conversion
The Significance test (http://getdatadriven.com/ab-significance-test) shows that the testing was significant.
Obviously not all clicks on the "buy now" button convert to an order. The other thing is what if 100% people ordered from B but only 5% from A? By looking at the clicks I don't know.
Of course if I look at my statistics I can guess how many orders did each variant generate.
My CTR before testing was 2.09% and my conversion rate (based on orders) was 0.55%. With a bit math I can then "assume" the following:
A: 556 visitors - 6 conversions (clicks) - 1.07% conversion B: 543 visitors - 3 conversions (clicks) - 0.55% conversion
The significance test however indicates that test was not significant, even though the ratio of conversions are about the same (A converting 50% better than B).
So using clicks seems like scientifically more significant if the number of orders are small.
Should I redo the test based on order as conversion metric? Shall I use order metrics for future tests?