12

Currently doing UX testing on low budget. We have prototypes in Figma and user tests in Maze.
Now one of the main goals is to compare two versions of a feature.

The problem?

screenshot of the software Maze

All the remote testing tools I've tried don't offer the possibility of randomization of tasks. So there will always be a strong learning effect for each user, because they start with A and go to B.

Solution Ideas

So what can I do?

  1. Add a general "intro task" that lets the subject get to know the user interface & context
    (resulting in unnecessary length of the test)

  2. Create two user tests and handle the randomization myself and send each person the correct link individually
    (resulting in annoying manual work & having to aggregate the results from 2 tests)

  3. Find a magic tool that somehow handles this randomization for me and gives me one single share link
    (still manual work of aggregating results)

Does someone have experience with this or has solved this problem before?

0

2 Answers 2

14

Well, as you say, the obvious answer is to randomize (there are other, more sophisticated ways to do this, but probably too much for what you're looking for).

If you use the same pool of participants and show 2 versions in a row, there is no way to avoid stimulus exposure biased results. for this, you need to use a technique called Counterbalancing

The following is a translation of a Psychology Research Methodology Manual in Spanish (couldn't find a similar one in English, it's possible counterweights is not the proper translation)

Counterweights or counterbalances

It is used mainly to control the effect of progressive order or error, and is based on the existence of a linear relationship between progressive error and the order each treatment occupies within the experimental sequence. As the ordinal number increases, the level of progressive error increases. Each subject is presented with the experimental conditions in a different order, so that in the total set of conditions in the experiment, each of them has been applied the same number of times in the same order.

Intrasubjective counterbalancing

It controls for progressive error at the individual level by ensuring that each subject receives the conditions or treatments in a particular order first and in the reverse order second. If we have three ABC conditions, they are ordered as follows: ABCCBA. The order ABC is applied to the subjects first and then the order CBA.

The role control technique distributes the progressive error evenly throughout the series of trials.

The intra-subject balancing technique has the disadvantage that each subject receives each treatment more than once, which increases the experiment time. To overcome this inconvenience, the following control techniques can be used.

Intragroup counterbalancing

It consists of administering different treatment sequences to different subgroups of subjects. With this technique, the effect of progressive error is controlled in the group and not formed individually as in intra-subject counterbalancing. Full Intragroup Counterbalancing

In this case, all possible permutations of the treatment sequences must be used. To calculate these, the factorial of the number of conditions is found.

Subgroup 1 AB sequence

Subgroup 2 sequence BA

If we have three experimental conditions ABC, the number of sequences is

n! = 3! = 3 · 2 · 1 = 6

ABC, BCA, CAB, ACB, BAC, CBA

But just in case you want to do it anyway: If you have some JS /PHP knowledge, you could just create a page and serve a random version and save the results to a DB. You could even record the response time. If you know Python you could also use PsychoPy, there are some templates to do this.

Otherwise, it's best if you separate the groups. One will see version A, the other version B, which is classic A/B testing. Because if you use the same pool, you have to use at least ANOVA, and that's a lot more work than doing a little manual work.

All in all, I would opt for splitting the groups, it's much less effort and the results will be more accurate

7
  • "you could just create a page and serve a random version and save the results to a DB" damn that's exactly what I've been trying to find on Google, it sounds so simple that I thought there must already be a tool for it. Put 2 links in, get one share link out. Then participants get one of the links randomly (50% chance).
    – Big_Chair
    Jun 23, 2021 at 11:31
  • Otherwise interesting points, thank you for sharing
    – Big_Chair
    Jun 23, 2021 at 11:32
  • Nice resource! Do you want to include the source as well, or name of the publication? Jun 23, 2021 at 13:50
  • @Big_Chair: Be careful with randomly showing links. 50% chance is exactly that, chance! If you are really unlucky you could end up with all users testing the same feature and nobody testing the alternative. Obviously that becomes less likely the more users you have, but do you really want to risk a 75/25 split for example?
    – musefan
    Jun 23, 2021 at 14:57
  • 1
    @musefan Yeah I understand what you mean, a pure 50/50 randomizer would be too stupid to work for this. I'll have to try different methods to see what works best.
    – Big_Chair
    Jun 23, 2021 at 15:10
2

My first thought would have been to set up two test groups: One starting with A the other starting with B just as you stated in your possible solutions. With the results of both groups combined, we can probably assume, negates the learning effect.

I'm also not certain that randomizing the tasks would yield an accurate result (80% starting on A could still happen randomly i suppose?), think sequential alternation of start task would give you a more balanced 50/50 view, of course this becomes more complicated when you have more than 2 tasks (A/B/C...).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.