4

I am new to A/B testing, i have looked at some sites for more understanding on what it is:

We are going to do A/B testing on our site using on of the tools available. This is our first time doing it so we thought it is better to ask from experienced/expert folks like YOU GUYS!

Is it necessary to only do this when site is LIVE? what's the best approach to A/B test for new features before making it LIVE?

Improve this question
1
  • I have edited your post so that it is only one question. As a Q&A site each post needs to be a single question that can be answered. Not only that, but your other queries are a bit too broad to work as questions on this site, so I have narrowed it down to the one most suitable.
    – JonW ♦
    Feb 24, 2016 at 9:30

3 Answers 3

Reset to default
2

First to explain what A/B test is.

Say you have a website where the homepage is really important. While you design it, you wonder whether it would be better to write in the heading section ( h1 tag). You say it would be better to write "Welcome to our website" while your colleague argues that "Take a look at our services" will do a better job.

To answer which heading is better you design one version of the homepage with "Welcome to our website" and one version with "Take a look at our services". Lets name them version 1 and version 2.

A/B test of heading

To get the answer you present to 50% of the real visitors version 1 and 50% see version 2. If the goal of your website is someone to subscribe to a billing plan you start tracking for each version how many subscriptions there are. After one month you watch the results and see that version 1 had 1000 visits and 12 subscriptions while version 2 got 1000 visits and 26 subscriptions. The second version gave you more money so you delete version 1 and use only version 2.

1) Even the smallest organization can do A/B testing, it doesn't matter the size of the company, all that matters is how much real traffic you got. The more traffic you have the faster you will get the results from A/B test. If you have for example 1000 visitors per month and test two variations only you will need approximately 1 month to get some results. This is because you need to reach statistical significance of 95%. To put it simply, this means that you will be 95% sure that the results are valid and true.

2) It's absolutely necessary to do A/B test with real traffic. Otherwise the results won't be valid.

3) According to my experience the first thing you should test is the heading text of your web pages. Then you can test layout, and other stuff.

4) Yes

You will probably have other questions, don't be afraid to ask, I'll be happy to answer.

Improve this answer
4
  • 3
    I would disagree with 'the first thing you should test is the heading text' as well as with 'don't test button colour as this won't give you much difference'. The whole point of AB testing is to confirm if this is true or not for that specific situation. Just because your test didn't give much difference doesn't mean it's the same for everyone else (for example I've conducted tests around button colour that very much do cause a difference in conversions).
    – JonW ♦
    Feb 24, 2016 at 10:24
  • Ok, you are right you never know what will give you lift in conversions. I've edited the answer.
    – Kristiyan Lukanov
    Feb 24, 2016 at 10:26
  • 1
    Also it not right to assume that with 1000 visitors you'll get a 95% statistical significance, as long as you don't use the difference of conversion rate. Using this simple calculator getdatadriven.com/ab-significance-test we can clearly see that on an AB test with (varA, varB) having (1000, 1000) visitors and (100, 110) conversions it's not statistical significant.
    – gpelelis
    Mar 2, 2016 at 8:18
  • Yes you're right but I've used the word approximately. There are a lot of variables that can influence when significance will be reached, so I did a rough approximation.
    – Kristiyan Lukanov
    Mar 2, 2016 at 10:03
1

Yes, you can run an a/b-(N) test before it's live. However, statistically speaking you may need more samples to prove which solution works better. In manual test set-ups, finding participants can turn to be very expensive. For that reason, using the current visitors as participant makes things more reliable and and reduces the cost to run.

If you don't have any traffic in your site, it is not going to be efficient to use these tools.

a/b-n test is a method and can be applied to anything, like logos. Here another tool that I am using for simple preference tests which you don't need to have a LIVE website.

https://usabilityhub.com/preference-test

Improve this answer
1

To answer your main question:

Yes, you must do your test when your site is live. On the other hand you can preview your test before it is launched.

So what to do before launch?
First of all run a A/A test for 1-2 weeks and see if there is any difference on your data. By the way A/A test is a test that both variationA and variationB are exactly the same.
But why start with an A/A test?

  1. If any variation has a big difference on conversions or visitors, then something is wrong with the implementation. You have to contact the support of the A/B testing platform or just try another platform the platform.
  2. You'll have run a "succesful" test and you will feel confident about the use and the data of the A/B testing platform.

Secondly decide wheter you'll use platform's editor, or you'll use a developer to do the a/b testing with custom code.

  • Use platform's editor to test easy changes like wording on homepage, or buttons and number of fields on checkout.
    That kind of changes are easier to do and QA
  • Use a developer to write cutom code if you plan to "big changes" like testing new layouts for your website, or completely changing a page. This type of test are harder to do, and will need a lot of QA

Third just use the preview mode that each platform provides, and be sure that everything works correctly.

What to consider on your QA:

  1. Flashing variation (is the situation when your visitors notice your original version of some miliseconds and then they see the variation) - some solutions to flashing
  2. You don't break any behaviour (for example on older browser). It's better to exclude some visitors from the test, instead of serving a broken experience.
  3. Tracking is working well (you may keep an eye of the difference between the #unique-visitors reported by the a/b testing platform, and the #unique-visitors from google analytics).

Well I really suggest to read this QA & Troubleshoot from optimizely before launching. The same principles outlined there, also apply to each platform.

Lastly develop a strong knowledge of how a/b testing algorithm works, and when is the right time to run a test (in terms of traffic availability)

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.