A/B Testing Homepage Variants

Question

When trying out new homepage layouts / styles, is A/B testing the best way to do this?

With typical A/B tests you have a specific conversion measurement (i.e. version X drives significantly more people to click the 'sign-up' button). However with a homepage there isn't a single specific conversion; you're just looking to see if the visitor does anything from the homepage other than abandoning the site (clicks a menu option, clicks to read a news article, selects a featured product...)

What sort of testing should be used for homepage redesign if A/B isn't appropriate? How do you know if one version of your homepage is 'better' than another?

Answer 1

Without more information about the type of business you're in, it's hard to be specific, so I'll be broad. If your website is selling something in any way, you should be measuring sales by that recipe in aggregate, not individual actions taken from the homepage.

If you sell items directly, then you would measure the average dollar amount sold per unique visitor who was part of that recipe; if you sell display ads that go alongside content then you could use something like the total number of pageviews as a proxy for advertising revenue; if you are selling consulting services and the website generates leads, then you measure the number of qualified leads.

Measuring bounce rates makes a certain amount of sense, and it would certainly be easy to implement, but it's always best to have your metrics be the core aspects of your website's business model. A hypothetical example of where an indirect metric like bounce rate can go wrong is that a confusing or misleading homepage might lead users to get one click further in before they decide they're in the wrong place. The metric would look good, but it might just be getting the wrong customers in further where they end up not converting.

Answer 2

I disagree completely that you should be measuring a bounce rate on the home page. What's the thing that brings you revenue? Or the thing you actually want the user to do (submit a comment, blog post, etc)?

Bounce rate is a useful statistic but it should never be your single conversion goal. Let's take an ecommerce example in which you are only measuring bounce rate.

It could be with a new redesign that the bounce rate actually gets higher because the content/messaging is more clear, meaning that the people who don't need it figure out they don't need it faster which equals higher bounce rate. At the same time the people who do need the product actually figure that out faster which results in a higher conversion rate to the actual sale.

In this hypothetical you would track the "receipt" page in order to show a sale conversion. Its entirely plausible that the bounce rate on the home page increased but revenues also increased. I'll take the revenue increase every time.

Moral of the story: Always track what the most important action is for your website. Clicks and bounce rates are good statistics to have but are fairly useless in an a/b test setting.

Answer 3

A/B can be a good way to test 2 layouts with identical copy and content, but after you find the 'winner', multivariate tests where you try a selection of messages or calls to action might let you understand what drives customers to do something most effectively. You could experiment with different tones of voice, or evoking different emotions - it depends on your audience and how you think it's best to speak to them.

Answer 4

I have decided that the best course of action here is to still stick with A/B testing, but to check for bounce-rates and drop-offs rather than to test for conversions.

The homepage designs do go through several iterations before actually being built, so in those early stages we will get as much user feedback as possible before adding it to the site, and using a combination of Google Web Optimizer and Google Analytics we'll monitor the bounce rates of the new / existing homepage design. This, compared alongside the existing known bounce rates of the homepage are probably the best way to go.

It's a bit of a risk rolling out a new homepage design to 50% of the audience when we may very well not go for it as the final design, this could cause confusion to those users lucky/unlucky enough to have been presented with the new design only to lose it a week or so later. However it's better to test something out and then discard it than to roll it out permenantly and just hope people like it.

Answer 5

Now it's not about the tool, it's about the pattern you're going to use, first of all, you have to create personas, if you have at least 1 solid persona you would know better what to change in each variation, it's better to make 1 change per variation to be able to track what works better and then add another enhancement, for example, in variation 1, change the color of the explanation text or the "what is your website" text, in variation 2 increase the size of the same text with its control color, when 1 of them wins, make another change and test again, because 2-3 changes per variation would limit your insight, you might have a winning variation with 2-3 changes but you'll never know which (of the 3 changes) led to this win. so you have to set a persona, choose a tool, set the targets. logically, you're going after conversions, and decreasing bounce rate, these are the 2 goals you have to set before getting started, and for the persona, let's say your audience is 18-25 males who care about cars and mid-day hangouts, you might need to experiment with making a variation with an attractive image above the fold with really nice looking call to action button, and in variation 2 you try no image but a great emotional text with really evident signup/login links, now when 1 of these win you make 2 variations of it, and so on. now for the tool to use, try Visual website optimizer, visualwebsiteoptimizer me and my team use this one and we're still experimenting with it. it allows you to test many variations live and divide the percentage of users between them, you just have to add a code to your database. It works for testing homepages and any internal page as long as you have perfected URLs. What other teams use is google website optimizer which is a handy tool to test homepages; it provides A/B experiments and multivariate experiment which compares the performance of content variations in multiple locations on a page. Let me know if it worked for you :)