Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I have been reading and learning a lot about A/B testing and I get the process. But there are a few concerns which make it challenging.

  1. You need a lot of visitors to get statistical significance.
  2. You can only test one change at a time so that you isolate it.
  3. You have to have some degree of knowledge to know what to change and how to engage users.

In my case, I have a template that I use and I want to do these tests BEFORE many people come to the site so that they don't all just leave. So I am not sure what might be the best path to proceed.

Any advice on what I should do? Right now I just want to get people to participate in the site more after they find it.

share|improve this question

migrated from answers.onstartups.com Aug 29 '12 at 20:17

This question came from our site for entrepreneurs looking to start or run a new business.

    
@JonnyBoats good call! Didn't realize that existed...will try it :) –  Genadinik Feb 6 '12 at 19:47
    
I second JonnyBoats. Or try JOS Biz forum where people like Patrick McKenzie show some info about A/B testing. –  Ross Feb 7 '12 at 7:51

1 Answer 1

up vote 2 down vote accepted

You need a lot of visitors to get statistical significance.

Yes you do - no way around this basic statistical law. However if you make big changes where one outperforms another dramatically then this will reduce the number needed. Smaller changes where the results are practically identical will run forever.

Big changes also help you get over the local minima problem.

You can only test one change at a time so that you isolate it.

Most of the A/B testing tools let you do multivariate testing that will test a combination of changes at the same time but this leads to an increase in traffic needed to come up with a valid result and can be more complex to setup (e.g. you will get in a mess if some combinations of the things your testing are 'invalid' or dont make sense to a user)

You have to have some degree of knowledge to know what to change and how to engage users.

ALL tools require some degree of knowledge to use effectively ;) For some inspiration see :-

In my case, I have a template that I use and I want to do these tests BEFORE many people come to the site so that they don't all just leave.

Perhaps then (guessing you don't have a lot of traffic) what you need to do is start off with some basic Usability Testing where you simply watch a few people try to use your site - you will find that a just a half dozen tests will unearth the real big problems.

The book "Don't make me think" is an excellent resource for this subject (and general site design) and there are a few sites out there that will help you do these tests very cheaply such as UserTesting.com. (no affiliation)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.