A/B Testing Home Page of an Ecommerce Site

Question

I am very new to any sort of website testing (please be gentle!), and I have some questions (mostly just for my own understanding) about A/B testing an ecommerce site's home page.

The company I work for wants to run two A/B tests on the website's home page at the same time:

Test 1

• Targeted to customers who we believe (based on spending habits) are beginners in the hobby our products are used for
• 50% of these customers will see a control group of products
• 50% of these customers will see a group of products targeted at beginners

Test 2

• Targeted to customers who we believe (based on spending habits) are intermediate to advanced participants in the hobby our products are used for
• 50% of these customers will see the same control group of products specified above for Test 1 (although I believe we would treat them as a separate group)
• 50% of these customers will see a group of products targeted at intermediate to advanced users

For both tests, the plan is to monitor conversion rate for the four different groups. Although I have not been told, I assume the goal of the test is to decide if displaying products targeted to the user's level of experience in the hobby will improve overall conversion.

Questions for my own understanding:

1. Does the "distance" on the path to purchase between the home page and the result we are measuring (an order) affect the validity of the test?

2. Would there need to be at least general consistency in each test (not across the two tests) in the pricing and offers associated with each test? For example, what if Test 2 has a control group of products that averages a price point of $50 and a testing group of products that averages a price point of $1,000 - won't that sku the results to the control group of products? Another example: what if Test 1's testing group of products is dominated by skus which have a free shipping offer attached to them while the control set in Test 1 does not?

3. My very rudimentary understanding of testing is that you won't reach statistical significance until you have 100 orders (I was told this by a marketing executive in the print catalog industry who did not have a huge amount of experience in the online industry - they told me this was a standard understanding in direct marketing).

If that's the case, would you think the test has reached statistical significance when there have been a total of 100 orders, when there have been a total of 100 orders of the products displayed on the home page to the various groups, when each group has placed 100 orders, or when each product has been ordered 100 times?

I'm guessing that at least some of my questions reflect my ignorance, but I'm hoping that someone will be able to help me understand.

Thanks in advance! Any help anyone can offer is much appreciated!

Answer 1

1. No, it doesn't matter how many steps are in between because you are testing two alternate universes where the only difference is the test. If you run a test between apples and oranges, and find out that it makes people buy pears, you can conclude that the results are either true or you didn't design your test correctly... which leads us to - I'm going to update my answer here to say that there are some cases where leakage in the time domain will screw up your results. If, for instance you have a cart that allows people to store items between sessions to buy later, then it's possible that some of your results will be thrown off by the fact that people are returning and buying items that they had decided to buy previously.

2. Yes. You need to hold everything as constant as possible except for one variable. If you want to decide whether intermediate users are more likely to buy intermediate items, you have to hold price constant, or as constant as you can. If you were to run the flawed test above, and your new version succeeds, the only thing you can take out of your test is:

   • intermediate users like intermediate items, or
   • they are more/less likely to buy more/less expensive items, or
   • both.

3. This "100 item" rule is totally, horrifically wrong. There is a lot of statistics behind determining significance, but you can use a significance calculator to vastly simplify your job, like this one: Thumbtack ABBA split test calculator. There's lots of explanation in the linked page, but, basically, your P value will determine whether your test is significant or not and you will be looking for P value of probably .05 or below. The confidence interval is completely separate – you choose the interval (the default is 95%) – and the confidence interval is used to calculate the confidence spread that you find in the last column of the results. In any case, 100 successes is certainly going to be much too low.

Also, I'll include a quick tip. Sometimes, your tests will converge to significance faster than they really should. Remember, it's still all statistical. Resist the urge to stop your test as soon as your P value drops below .05. Let it run a little bit longer.