What is the point behind an A/B test if everything is decided by users' feedback on a website? Is there any role for a UX guy in this case?
Should we give preference to the results of A/B testing over what a UX guy suggests to increase traffic and usability of websites?
A/B testing is very limited in scope; it is concerned only with single specific measurable conversions. You can't A/B test for Satisfaction, Enjoyment, or User Experience. The 'UX Guy' is responsible for the whole user experience and isn't solely focused on a single narrow objective.
You also need somebody to determine what to A/B test. That would come to your 'UX Guy'. You shouldn't just be A/B testing anything randomly if you want to achieve a usable website with the best possible return; the UX Guy should identify possible areas that could be improved based on their own expert knowledge ("this button should be given an arrow to indicate that it is progressing further into the application") and the set up the AB test to confirm / deny that test.
If it were left to a marketing person then you'd possibly end up with the an AB test look at whether a large rotating button with the company logo is better for conversions than a Flash animation button that plays a noise when you hover over it. It's very possible one of these options will perform better than the other, but that doesn't mean either of them are the right option. In-fact a better test would be to check if changing the wording on one button converts better than another, and the UX Guy would determine if this is so.
UX designers play tremendously important role in both test design and analysis of results.
During the test design stage, they make sure that users have the choices "good", "better", and "best" instead of "the lesser of X evils".
When it comes to analysing the results, UX designers aren't looking just at one screen but at the big picture of the workflow. They consider what happens before and after the testing point to make sure that qualitative analysis supports the quantitative results. They may find out that the test holds valid only for very particular situations, which will take them back to the drawing board.
Well, without being able to personally prequalify your UX consultant, there's a reason (usually) why they're hired, which is to direct the user experience process. Part of that is (again, usually) utilizing A/B testing.
You shouldn't have to choose between the two. Put your UX input to the test by using A/B testing and other tools. Gauge that against what you are currently offering. Make more improvements. Rinse and repeat continually, and hope you don't stop listening to what your users are telling you.
Both usability and A/B testing are tools at your disposable. Both are important and should be used as part of your larger toolset, which includes other things such as heuristic evaluation, market research, etc.
Usability will, to some extent, always be suppositional or clouded or skewed; whatever term is appropriate. There's often a lot of set-up and explanation associated with usability, which is why we write test backgrounds...to bring the participant up-to-speed on the scenario and the test. Then we say things like "be candid in your feedback" and "you won't hurt my feelings".
By trying to set the context of the test with the user, you're already asking them to imagine a scenario and they will begin to project attitudes and responses, based on what they imagine. And then there can be bugs in the prototype. Because of these issues, your findings may be directionally accurate but will never match a true user in their true environment in your production app.
Production testing is the only way to truly measure the effect of the product. A/B testing is for fine tuning the effect of a nuance of a product. A/B and multi-variate testing can be used for lots of reasons, so you'd be best off researching the correct test type.
