Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

When trying out new homepage layouts / styles, is A/B testing the best way to do this?

With typical A/B tests you have a specific conversion measurement (i.e. version X drives significantly more people to click the 'sign-up' button). However with a homepage there isn't a single specific conversion; you're just looking to see if the visitor does anything from the homepage other than abandoning the site (clicks a menu option, clicks to read a news article, selects a featured product...)

What sort of testing should be used for homepage redesign if A/B isn't appropriate? How do you know if one version of your homepage is 'better' than another?

share|improve this question
    
I think you need to identify some sort of goal or goals. It's the only way you're likely to get a picture of what works and what doesn't. When doing your designs, have you not had particular things you want people to do? Do you not have blocks or areas which have been done in such a way to lure users/visitors into choosing? –  Sam K Jun 27 '11 at 10:19
    
@Sam, I think that's what the problem is. There isn't a specific goal. For an eCommerce site the homepage shows products, news, company info, specific 'featured' products... There's too much on offer to do a valid AB test. It doesn't aim to get visitors to anywhere specific but to give an overview of the site and what's available to the user. I'm thinking maybe just testing BOUNCE RATE may be the best option here, not versions that perform the best, but those that are the 'least worst'. –  JonW Jun 27 '11 at 10:39
add comment

5 Answers

up vote 3 down vote accepted

Without more information about the type of business you're in, it's hard to be specific, so I'll be broad. If your website is selling something in any way, you should be measuring sales by that recipe in aggregate, not individual actions taken from the homepage.

If you sell items directly, then you would measure the average dollar amount sold per unique visitor who was part of that recipe; if you sell display ads that go alongside content then you could use something like the total number of pageviews as a proxy for advertising revenue; if you are selling consulting services and the website generates leads, then you measure the number of qualified leads.

Measuring bounce rates makes a certain amount of sense, and it would certainly be easy to implement, but it's always best to have your metrics be the core aspects of your website's business model. A hypothetical example of where an indirect metric like bounce rate can go wrong is that a confusing or misleading homepage might lead users to get one click further in before they decide they're in the wrong place. The metric would look good, but it might just be getting the wrong customers in further where they end up not converting.

share|improve this answer
    
That is a very good point; it is always possible that the visitor will try out at least one link before abandoning the site, so the bounce rate is not a perfect test. That's something I'll need to consider when reviewing the results. It is not directly sales that are measured on this site, a lot of the visitors come purely for research before purchasing something, so we'd be looking to find out if the site design helps them with this. That's why finding a specific measurement to AB test is tricky from a page in such a high level of the hierarchy. There's lots of options for them to take here. –  JonW Jun 30 '11 at 7:53
    
I see - it's always very tricky to use metrics when the sales are indirect. I don't envy you :) –  Jonathan Jun 30 '11 at 18:40
add comment

I disagree completely that you should be measuring a bounce rate on the home page. What's the thing that brings you revenue? Or the thing you actually want the user to do (submit a comment, blog post, etc)?

Bounce rate is a useful statistic but it should never be your single conversion goal. Let's take an ecommerce example in which you are only measuring bounce rate.

It could be with a new redesign that the bounce rate actually gets higher because the content/messaging is more clear, meaning that the people who don't need it figure out they don't need it faster which equals higher bounce rate. At the same time the people who do need the product actually figure that out faster which results in a higher conversion rate to the actual sale.

In this hypothetical you would track the "receipt" page in order to show a sale conversion. Its entirely plausible that the bounce rate on the home page increased but revenues also increased. I'll take the revenue increase every time.

Moral of the story: Always track what the most important action is for your website. Clicks and bounce rates are good statistics to have but are fairly useless in an a/b test setting.

share|improve this answer
    
Sure, but testing the final actions that a user does further into the site is not directly related to the homepage. The site as-is will still exist beyond the homepage, so any issues with purchasing / commenting etc once they have moved on from the homepage will be related to those pages themselves, not really the homepage. Aside from getting user feedback and expert opinions on homepage design actually getting feedback on the live homepage itself is the problem. How do you measure if a homepage design is successful? –  JonW Jul 1 '11 at 9:07
    
The homepage is the very "top of the funnel" for your website. So while the pages after the homepage affect the traffic on those page, the homepage is the gateway that gets them there and also sets the tone for the rest of the site. –  Msencenb Jul 1 '11 at 15:53
    
Measuring the success of a design should ultimately be related to the conversion goal no matter if its deep into a site or not. The homepage ushers users into a "funnel" in which your conversion goal takes place so the success of the redesign depends on your bottom line still. A successful redesign will end up with more purchases/whatever because it pushed more people into the funnel. Measuring bounce rate or other stats won't tell you this statistic it will just give you one piece of the puzzle. –  Msencenb Jul 1 '11 at 15:57
add comment

A/B can be a good way to test 2 layouts with identical copy and content, but after you find the 'winner', multivariate tests where you try a selection of messages or calls to action might let you understand what drives customers to do something most effectively. You could experiment with different tones of voice, or evoking different emotions - it depends on your audience and how you think it's best to speak to them.

share|improve this answer
add comment

I have decided that the best course of action here is to still stick with A/B testing, but to check for bounce-rates and drop-offs rather than to test for conversions.

The homepage designs do go through several iterations before actually being built, so in those early stages we will get as much user feedback as possible before adding it to the site, and using a combination of Google Web Optimizer and Google Analytics we'll monitor the bounce rates of the new / existing homepage design. This, compared alongside the existing known bounce rates of the homepage are probably the best way to go.

It's a bit of a risk rolling out a new homepage design to 50% of the audience when we may very well not go for it as the final design, this could cause confusion to those users lucky/unlucky enough to have been presented with the new design only to lose it a week or so later. However it's better to test something out and then discard it than to roll it out permenantly and just hope people like it.

share|improve this answer
1  
This should work, bounce rates are the most significant here, especially for mainpages, we released a new design for our home page a month ago, after we collected so many feedback and tested every single details, we released it by, creating an renouncement bar on the old mainpage that goes down when you enter the site says: "we've released a new design, try it now!" so this way, your users will have the option wither to use the new page or stay on the old one in the early stage, we removed it a month later, 80% or our users were already using the new design. gradually, it works :) –  Motaz Al-Thaher Jun 28 '11 at 22:06
add comment

Now it's not about the tool, it's about the pattern you're going to use, first of all, you have to create personas, if you have at least 1 solid persona you would know better what to change in each variation, it's better to make 1 change per variation to be able to track what works better and then add another enhancement, for example, in variation 1, change the color of the explanation text or the "what is your website" text, in variation 2 increase the size of the same text with its control color, when 1 of them wins, make another change and test again, because 2-3 changes per variation would limit your insight, you might have a winning variation with 2-3 changes but you'll never know which (of the 3 changes) led to this win. so you have to set a persona, choose a tool, set the targets. logically, you're going after conversions, and decreasing bounce rate, these are the 2 goals you have to set before getting started, and for the persona, let's say your audience is 18-25 males who care about cars and mid-day hangouts, you might need to experiment with making a variation with an attractive image above the fold with really nice looking call to action button, and in variation 2 you try no image but a great emotional text with really evident signup/login links, now when 1 of these win you make 2 variations of it, and so on. now for the tool to use, try Visual website optimizer, visualwebsiteoptimizer me and my team use this one and we're still experimenting with it. it allows you to test many variations live and divide the percentage of users between them, you just have to add a code to your database. It works for testing homepages and any internal page as long as you have perfected URLs. What other teams use is google website optimizer which is a handy tool to test homepages; it provides A/B experiments and multivariate experiment which compares the performance of content variations in multiple locations on a page. Let me know if it worked for you :)

share|improve this answer
    
I think we'll go for the GWO and GA route and just check bounce rates. The idea of setting personas may be useful for deciding what to go in the homepage, but I'm more concerned about how to test the homepage once all those decisions have actually been made and it's actually been produced. –  JonW Jun 28 '11 at 10:32
    
How about you give usabilla.com a try, it's not an A/B testing tool but it's an effective remote usability testing tool, if you have usability problems, when you identify them, you might know how to create variations. –  Motaz Al-Thaher Jun 28 '11 at 12:47
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.