I want to do some test-and-learn on a multi-step form using infrastructure which appears to support conducting and reporting of A/B/n tests.

If my site had two different forms I could simultaneously run A/B tests on each form and calculate the statistical significance of each test totally independently of the other.

On the other hand if I simultaneously tested two (discrete) variables on the same form, against the same success factor, I would be conducting a multivariate test and would need larger numbers to achieve a suitable confidence level in my results - assuming the system doesn't do this for me, I plan to use a chi-squared test.

But what if I have a two-page form, and I test one variable on Page 1 and another on Page 2? Can I simply say that each test has a separate success factor (completion of Page 1 v. completion of Page 2) in which case I can treat it as two parallel A/B tests, or would it be safer to say that they both have the same ultimate success criteria (completion of the final Page, 2) so that they should be treated as a multivariate test?

  • 2
    Think you have to consider the users. For them it is one form, so they will have set in their minds on page 2 what they saw on page one and will see inconsistencies across, not separate factors. Doesn't this then carry a pre-bias into page 2? – Susan R Jul 12 '10 at 14:39
  • Good point. I need to be aware of possible inconstancies between say, styling or page layout variables on different pages. In fact I am thinking mainly of content variables which I will make it my business to ensure have no obvious inter-page affinities. I suspect I am just going to have to try this and see if subtle affinities emerge - but I'd love to know if anyone has already trodden this path... – Francis Norton Jul 12 '10 at 16:18

Yes, performance on Page 2 could depend on the test condition of Page 1, but analyzing them as separate tests is no worse than most use of A-B testing. We conduct iterative design where we do an A-B test on Page 1, implement the change, then go on to do an A-B test on Page 2. The only difference is that you’re doing the two tests simultaneously rather than serially. That doesn’t change the underlying assumption that the performance on Page 2 is independent of the design of Page 1, nor does it appreciably change your statistics (your could argue that there’s a small increase in the family-wise error rate, but that’s all that comes to mind). The statistics are valid as long as you run each change as separate tests and don’t act like you’re doubling your number of users because you’ve two datapoints per user.

For that matter, the statistics would be valid even if you were testing two changes on the same page. Again, that’s not appreciably worse than serially testing two changes for the same page which is common in iterative A-B testing.

Potential dependence of Page 2 performance on Page 1 design isn’t a statistical issue. Instead, the risk is that you may not considering all the variables you should to get the best design. In principle, iterative design with A-B testing can result in a design optimized to a local maxima. The same thing can happen with iterative design that employs usability testing, but usability testing generally means you have qualitative data that provides insight on dependence.

Practically, I would only expect potential dependence to be an issue when there may be inconsistency. For example, suppose you’re testing on Page 1 whether to indicate locked fields with a gray background or a red border, and you may find the red border works best. On Page 2 you’re testing whether to indicate required fields with an asterisk or a red border, and you may find on average a red border works best. However, if you were to look at users who had a red borders for lock fields on Page 1 and red borders for required fields on Page 2, you’d see they perform the worse of all the combinations due to the inconsistent use of red borders. Whether you do the two separate A-B tests in parallel or serially, you never discover that the best overall performance is to use a gray background for Locked and a red border for Required.

In your case, I don’t think you need to decide ahead of time whether to run it as a single omnibus test or two parallel A-B tests. Run the test long enough to get the sample you need for separate A-B tests and run the analyses, but also construct your 4x2 1A2A-1A2B-1B2A-1B2B contingency table and look at the descriptive statistics to see if there’re signs of dependence that would impact your design decision. If there is, let the test run longer to gather more data then do the inferential statistics. That’s something you couldn’t do with iterative A-B testing.

  • Thank you - a great answer, worth reading slowly and digesting. – Francis Norton Jul 15 '10 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.