3

I am running some usability tests in the coming days. I have three options of one feature for a pretty simple and quick test.

My concern is that because each of the options are very similar to each other, once the user understands how the first one works the other two would be easy to figure out.

I will be rotating the options so as to not start with the same test each time.

My goal is to find out which option makes the most sense and is easiest for them to complete a particular task.

Any advice would be appreciated.

  • The short answer: you don't stop it. That's why it is a problem. – Evil Closet Monkey Jun 6 '18 at 15:53
  • You've made a good choice to rotate the options to avoid learnability between options. I think this is the quickest and cheapest way to get it done. – RobbyReindeer Jun 8 '18 at 7:28
2

When I run into these, I go back to "What am I trying to learn?"

In this case, I'd revisit how I'm going to measure what "Makes the most sense" and what is "easiest".

If you have the users self-rate (i.e. How easy was this on a scale from 1-5), you may run into users always thinking the last option was best/easiest ("recency bias") and users wanting to say what they think the researcher wants to hear.

If you're able to use something like Time on Task or Number of Taps, it might help bridge the Say->Do gap as you'll be measuring how long it actually takes them instead of how long they felt it took them. Often very different things. =)

Something else coming to mind is showing three separate groups of user testers only a single version of your design and compare stats between the three so you completely remove the "figuring out the other two" issue.

Once I worked with the team who handles the live website and had them target real users coming into the site. They were able to put a banner for certain visitors which linked to our User Test. We had to tread carefully as management people were rightly very wary of sending users into an area with an incomplete experience, but we were able to quickly get fast data from real people visiting the website (we were testing who would try to do something online through self-service instead of calling or emailing the company).

I always find it helpful to have someone outside my UX team do a "Dry run" of the test, as well as someone inside the UX team as they will likely be able to give really pointed feedback.

I hope some of this helps!

At the end of the day, you'll be learning something no matter what and the fact you're doing any User Testing is great news. Keep it up!

2

Evil Closet Monkey alludes to this but it's an effect that every research plan has to contend with. If you have enough people in there and are rotating the question order, it should mitigate that familiarity effect enough that it's a wash across all the participants.

It's also worth asking (I think Jason makes this point also maybe) whether or not you even need the two very similar tasks in the same usability experiment. Can you get your research answered question without employing 2 distinct but very similar options?

1

Only do one task per user. That way there can't be any interference of prior knowledge.

If you can't get enough users for that, definitely rotate the oder of the tests. You could also have the users only do 2 tests out of 3, each a different combination to further reduce the influence.

0

This is the classic Within-subjects vs. Between-subjects conundrum. Sounds like you're leaning toward doing a within-subjects testing approach. If you're concerned about knowledge transfer or "interference" as it is called in most HCI literature, what you could try instead is a between-subjects approach, which means each user is tested under one condition only. Only real disadvantage of between-subjects is you're going to require a larger number of participants to get useful data.

You could also try the counterbalancing/Latin squares technique for within-subjects, which is about presenting all options to each user but in different orders to essentially cancel out any interference. You're "rotating" options, so sounds like you're doing something similar but perhaps apply that rotation to a proper Latin Square Analysis of Variance to make it more sound.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.