I've two questions I would like to pitch to the community. I've done A&B testing previously but I'd like to check in with you to see if there are any methods I should consider using.

When facilitating the testing session with 5 participants, do you:

Q. Alternate the order in which you present A and B options?

Q. To make analysing your findings easier, you ask participant to score 1-5 how easy or hard the task was?

Thanks in advance.

2 Answers 2


To answer your first question: yes, you must change order. In psychology research this is known as counterbalancing:

A technique in experimental design that is used to avoid the introduction of confounding variables. In usability testing, this technique is most commonly used when establishing task order.

For example, a website user test might ask participants to complete two tasks: 1. find the sitemap, 2. sign up for the email newsletter.

If the study’s 12 participants complete these tasks in the same order each time, a researcher who observes high success rates on task #2 may falsely conclude that finding the email newsletter is easier than it really is. The reality may be that success rates on task #2 were artificially inflated because participants had already learned important things about the website as they completed task #1.

Counterbalancing the task order by having 6 participants begin with task #1 and 6 participants begin with task #2 removes this potential confound.

As for your second question, I have never asked how hard the task was in A/B testing, because that would be a whole different kind of test, it looks like you're trying to use a Likert Scale or similar in an A/B test.

It also sounds like the test is presential, which I'm not sure if it's forbidden in A/B, but I'm 100% sure that you'll need to take much more measuring controls to avoid bias and contamination of samples (counterbalancing is one of those controls, btw).

Don't know if this answers your question. If not, you'll need to describe and explain your methodology since it sounds like some kind of particular test

  • Thanks Devin, I see your point and yes, scoring the task isn't what A/B testing is about, its not needed. I didn't know that in randomising of the test options was known as 'Counterbalancing' and now I do. Thanks for your time.
    – user105669
    Commented Aug 19, 2017 at 19:57

This sounds like a preference test rather than an A/B test. If I've understood the question, you are presenting two variations to each participant in a session and asking for their feedback on each.

If you use one of the online tools (like UsabilityHub) they do tend to randomise the presentation of the variations in order to control for sequential bias, and also ask participants to quantify their opinions on a scale.

  • Its an A&B test for sure, the preference will be in users reaction. We have two different navigation options for the same page. I've not used the UHub tool. Might give it a look.
    – user105669
    Commented Aug 16, 2017 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.