I am very new to any sort of website testing (please be gentle!), and I have some questions (mostly just for my own understanding) about A/B testing an ecommerce site's home page.
The company I work for wants to run two A/B tests on the website's home page at the same time:
- Targeted to customers who we believe (based on spending habits) are beginners in the hobby our products are used for
- 50% of these customers will see a control group of products
- 50% of these customers will see a group of products targeted at beginners
- Targeted to customers who we believe (based on spending habits) are intermediate to advanced participants in the hobby our products are used for
- 50% of these customers will see the same control group of products specified above for Test 1 (although I believe we would treat them as a separate group)
- 50% of these customers will see a group of products targeted at intermediate to advanced users
For both tests, the plan is to monitor conversion rate for the four different groups. Although I have not been told, I assume the goal of the test is to decide if displaying products targeted to the user's level of experience in the hobby will improve overall conversion.
Questions for my own understanding:
Does the "distance" on the path to purchase between the home page and the result we are measuring (an order) affect the validity of the test?
Would there need to be at least general consistency in each test (not across the two tests) in the pricing and offers associated with each test? For example, what if Test 2 has a control group of products that averages a price point of $50 and a testing group of products that averages a price point of $1,000 - won't that sku the results to the control group of products? Another example: what if Test 1's testing group of products is dominated by skus which have a free shipping offer attached to them while the control set in Test 1 does not?
My very rudimentary understanding of testing is that you won't reach statistical significance until you have 100 orders (I was told this by a marketing executive in the print catalog industry who did not have a huge amount of experience in the online industry - they told me this was a standard understanding in direct marketing).
If that's the case, would you think the test has reached statistical significance when there have been a total of 100 orders, when there have been a total of 100 orders of the products displayed on the home page to the various groups, when each group has placed 100 orders, or when each product has been ordered 100 times?
I'm guessing that at least some of my questions reflect my ignorance, but I'm hoping that someone will be able to help me understand.
Thanks in advance! Any help anyone can offer is much appreciated!