Unexpected A/B testing results

Question

We've just run an A/B test for one of our landing pages using Visual Website Optimizer. We are an online hotel booking reservation site.

The landing page shows information about a particular hotel (pictures, description, map, amenities) and a search an engine box to select the dates of the stay, room type, currency... with a button to "search" availability on that property...

The search box is now placed at the bottom of the page (you have to scroll down to see it). The variation was to put the search box on the top of the page (horizontally in both cases). In this sort of landing pages, all our competitors have the search box on the top of the page (usually on the left hand side). The stats for the test: Percentage Traffic to include in Test: 100% Total visitors: 250

CONTROL (127): Submit Search: 34.65% Submit Booking: 4.72%

VARIATION (123): Submit Search: 27.64% Submit Booking: 3.25%

Contrarily to our hypothesis, visitors prefer to scroll down. They submit more searches and bookings (conversions) than in the variation.

Would you trust these results? Maybe no enough visitors?

Answer 1

What you've got there is a null result - there's no real difference between the two.

Let's backtrack from the percentages to actual numbers.

Control (127): Submitted search 44, booked 6

Variation (123): Submitted search 34, booked 4

Just by eye balling the numbers it's not looking terribly compelling. If just one person less in the control and one more person in the variation had booked we'd have the same numbers on both...

Now - there is this thing called statistical confidence. It's an expression of how confident we can be that we didn't just get the results through chance. This is often expressed as a "p value". It's generally accepted that you want a p < 0.05, which roughly translates as being 95% confident that the result is significant.

VWO even has a nice online tool to calculate it for you http://visualwebsiteoptimizer.com/ab-split-significance-calculator/ :-)

If we look at the p value for search it's 0.115 - not significant.

If we look at the p value for bookings it's 0.4 - not even close to significant.

Percentages are completely the wrong thing to look at with A/B testing. You need to be looking at the statistical confidence level that you have an improvement.

What you have here is a "not proven". You'd need to collect many more results, or have a more significant difference between results, before you could be confident that the change had made things better/worse.

Answer 2

Well. Half the idea of an A/B-test is to be surprised... ;-)

But I would definitely investigate this case more!

I wouldn't say that the findings are compelling, either. The difference is too small to conclude that your visitors prefer A over B (and you should have more users).

• Do some user testing or after task interviews to get more "quality information".
• Also find out why so few users use the search box at all. I would have thought that most users went straight to the search box...