A/B testing a site with low traffic

Question

I've always been very interested in A/B testing, but have never had the opportunity to implement it on any websites.

I work at a digital agency where the main reason why we haven't been able to implement A/B testing is that our sales guys believe that the website must have a massive amount of traffic for the A/B testing to pay off for the customer.

When is it worth setting up A/B testing? Is there some critical threshold of users/revenue at stake that should factor into my decision, or is it simple enough/important enough to start right away?

Answer 1

"Massive" amounts of traffic is a subjective term. Split testing doesn't necessarily need the traffic volumes of Google or Amazon to be effective. The key question is statistical significance, which you can achieve with low amounts of traffic provided the difference in results is great enough.

I built a quick and dirty split-testing calculator a while ago to help me test some PPC ad copy, but it's applicable to any test with two candidates (for anyone interested in how it works, the source explains more). If you play with the numbers you'll quickly discover what sorts of changes in your conversion volumes would be considered statistically significant.

It's often worth building in a framework to allow testing at the start, as part of your design phase. That way, you can add it as painlessly as possible and examine any other issues the testing might cause. Of course, you should only build it in if you expect to use it - and estimating whether it's worth it is the tricky part.

The key thing to bear in mind is that the larger the traffic, and the larger revenues, the greater the payoff from split testing. If the page drives revenues of $1,000,000+ per month, even a modest 1% improvement in conversion will result in an ongoing return of $1000+ per month.

There are a lot of case studies touted by the various companies that offer this as a service, and estimating from your own site is likely to be nearly impossible, so it's hard to pin down expected benefits. However, as an example, Mailchimp estimates that split testing email headlines can result in 11% increases in open rates. Testimonials from Visual Website Optimiser suggest returns from a 50% increase to 150%. Of course those are all likely to be the higher end of what's achievable, but the key thing to take away is that it can make a significant difference and depending on the cost of implementation might provide serious returns even on a smaller scale.

Costs to split testing are likely to vary with implementation, which makes them unfortunately difficult to estimate. I do know that Google provides a free solution through Analytics (Content Experiments). There's probably a range of other solutions depending on how much control you want - though which one you should choose is probably a question to ask on a different site, I suspect.