I’ve been doing a lot of thinking on the Halo Effect lately. It seems that this effect can really make a difference in the outcome of my testing if I am not cognizant of its potential swaying power. How can I minimize its effect during usability testing on a web application?
Only wear dotted shirts when testing ;-) .
As described (in Wikipedia), the Halo effect affects "the observer", but in a testing situation, there's an observer (or moderator) and a tester. The observer is probably you (if you ask this question here), but I assume you are worrying that the tester is looking at some "stripes" and will report "wrong" results.
My next question is: are you worrying about whether the testing situation (invitation, expectations, payment, facilities, moderator, browser etc.) will influence the outcome? Or are you worrying that aspects of the tested software will influence it, which you don't want to test (e.g. testing interaction while your visuals have not been finalized)?
To address the first, there's a standard procedure which reduces the effect. Pay adequate compensation, dress similar to your participants, clearly explain what you expect (i.e., "We are not testing you, but we you are testing this software" - don't say "our software"), etc. I have no reference, but you can certainly find a book or website detailing professional testing procedures.
To address the second, you must be very clear what you questions are. Before testing, list research questions. Good ones might be "Do users find the Login button?" or "Is the product description legible and relevant?", a bad one is "Do user like the site?". Then design tasks which allow you to observe the answers. Good tasks are "Please log in to the site (given account and password)." or "Does product XYZ have feature ABC?".
There are also standard questionnaires in circulation, which allow you to rate the user's overall experience, after the test. Questionnaire building is an art, don't try to invent one yourself if you're not into psychology and statistics. A few examples are here.
The best way to eliminate the Halo Effect is experience. Undoubtedly you will make mistakes when starting out, as you study testing through books and pages and posts such as this one, and most importantly - continue to do testing, evaluating you own performance along the way - the more you will become aware of influencing the test. This awareness is the best cure for the Halo Effect.