You need a lot of visitors to get statistical significance.
Yes you do - no way around this basic statistical law. However if you make big changes where one outperforms another dramatically then this will reduce the number needed. Smaller changes where the results are practically identical will run forever.
Big changes also help you get over the local minima problem.
You can only test one change at a time so that you isolate it.
Most of the A/B testing tools let you do multivariate testing that will test a combination of changes at the same time but this leads to an increase in traffic needed to come up with a valid result and can be more complex to setup (e.g. you will get in a mess if some combinations of the things your testing are 'invalid' or dont make sense to a user)
You have to have some degree of knowledge to know what to change and how to engage users.
ALL tools require some degree of knowledge to use effectively ;) For some inspiration see :-
In my case, I have a template that I use and I want to do these tests BEFORE many people come to the site so that they don't all just leave.
Perhaps then (guessing you don't have a lot of traffic) what you need to do is start off with some basic Usability Testing where you simply watch a few people try to use your site - you will find that a just a half dozen tests will unearth the real big problems.
The book "Don't make me think" is an excellent resource for this subject (and general site design) and there are a few sites out there that will help you do these tests very cheaply such as UserTesting.com. (no affiliation)
