Unexpected A/B testing results

Question

We've just run an A/B test for one of our landing pages using Visual Website Optimizer. We are an online hotel booking reservation site.

The landing page shows information about a particular hotel (pictures, description, map, amenities) and a search an engine box to select the dates of the stay, room type, currency... with a button to "search" availability on that property...

The search box is now placed at the bottom of the page (you have to scroll down to see it). The variation was to put the search box on the top of the page (horizontally in both cases). In this sort of landing pages, all our competitors have the search box on the top of the page (usually on the left hand side). The stats for the test: Percentage Traffic to include in Test: 100% Total visitors: 250

CONTROL (127): Submit Search: 34.65% Submit Booking: 4.72%

VARIATION (123): Submit Search: 27.64% Submit Booking: 3.25%

Contrarily to our hypothesis, visitors prefer to scroll down. They submit more searches and bookings (conversions) than in the variation.

Would you trust these results? Maybe no enough visitors?

Answer 1

What you've got there is a null result - there's no real difference between the two.

Let's backtrack from the percentages to actual numbers.

Control (127): Submitted search 44, booked 6

Variation (123): Submitted search 34, booked 4

Just by eye balling the numbers it's not looking terribly compelling. If just one person less in the control and one more person in the variation had booked we'd have the same numbers on both...

Now - there is this thing called statistical confidence. It's an expression of how confident we can be that we didn't just get the results through chance. This is often expressed as a "p value". It's generally accepted that you want a p < 0.05, which roughly translates as being 95% confident that the result is significant.

VWO even has a nice online tool to calculate it for you http://visualwebsiteoptimizer.com/ab-split-significance-calculator/ :-)

If we look at the p value for search it's 0.115 - not significant.

If we look at the p value for bookings it's 0.4 - not even close to significant.

Percentages are completely the wrong thing to look at with A/B testing. You need to be looking at the statistical confidence level that you have an improvement.

What you have here is a "not proven". You'd need to collect many more results, or have a more significant difference between results, before you could be confident that the change had made things better/worse.

Answer 2

Well. Half the idea of an A/B-test is to be surprised... ;-)

But I would definitely investigate this case more!

I wouldn't say that the findings are compelling, either. The difference is too small to conclude that your visitors prefer A over B (and you should have more users).

• Do some user testing or after task interviews to get more "quality information".
• Also find out why so few users use the search box at all. I would have thought that most users went straight to the search box...

Answer 3

With 250 users in an A/B test it’s fair to say it’s more than enough to trust your results. Jakob Nielsen wrote an infamous article in March 2000 saying you can capture 80 % of usability problems with only five users. The article Why You Only Need to Test with 5 Users, doesn’t mention A/B testing specifically but you can show it to your colleagues if there’s any doubt that 250 users in an A/B test is more than enough.

The conversion rates you get suggest you keep the search box out of focus for the user, under the fold. This doesn’t have to be bad. Maybe your site navigation in combination with the maps makes more sense to your users and they actually find what they want without searching? Elaborating on that, your search might performing badly driving users away. But we do not know that since I interpret the result as two different results. The only thing we do know is that test A (CONTROLS) have better conversion rates than B (VARIATIONS). That’s all you need to know – and stick with the design.

If you want to do more – I would suggest observations with 5 users (ref: Jakob Nielsen!) – giving them similar tasks on the different design A and B. You have to be present and looking at the screen when the users complete their tasks – and you’ll get the clues to why your result look the way they do. You see what users search for (if that’s the task) and how they behave. If they navigate – do they find what they want quicker than searching? If you have the equipment – use a video recorder and you’ll be able to let others see what happens, and make a more thorough analysis.

Do you need site search

Just found this great article after I posted, which further elaborates on the topic site search.

Searching for Better On-Site Search Usability:

(excerpts) Not every site needs, nor should have, an on-site search feature. But those that do must be sure that the search isn't just an after-thought. It needs to be more than something to add because you think visitors want it. Adding a search function is not necessarily good for on-site usability. Implementing a search function improperly is often a greater source of frustration than not having one altogether.