UX Comparative Test?

Question

I'm looking to do a comparative test on a prototype with 2 variations. The prototype is an onboarding strategy for a product. 1 version is designed with persuasive cues (social proof techniques) while the other has no persuasive cues. The prototype is tailored to each participant, so the participant will see actual data from their social circles.

I'm looking for ideas on how to evaluate the prototype. I'm trying to identify if the onboarding experience with persuasive techniques is more likely to encourage users to participate in the product quicker, than the variation with no persuasive techniques.

1. Are there any good models / tests for conducting this type of testing.
2. What type of questions could be asked in order to effectively evaluate the test?

I will be testing with 12 participants, each participant will review both variations. I plan on randomly selecting which variation a participant views first in order to help eliminate any bias with being exposed to a particular variation first.

Thanks!

Answer 1

In a perfect world you could test your prototype with dozens of people separated in two groups that will test either one or the other prototype. With large enough numbers both groups will be somewhat consistent.

With just 12 test subjects, testing both prototypes you need to take into account prior knowledge.

You'll need to determine clear metrics. When is the test or task a success and how can I measure this?
Maybe speed is a metric. Number of clicks. Completeness of profile information. Number of social connections made. Whatever your app is about.

I can't give a decisive answer because I don't know much about your prototypes, but you could either give the test subject a task or just let them browse while thinking out loud (also important during test with task!).

Avoid tasks and questions that are suggestive and be wary when asking for opinions (don't ask which of the two they liked better, let your test results speak for themselves).

I hope this helps (a bit).

Answer 2

You have a proposition that you want to test - that persuasive cues are more likely to encourage users to participate in the product quicker, than the variation with no persuasive cues.

This is classic A/B test, and you need to create a test scenario which avoids introducing any bias, including your own, and you need to be mindful of leading the participants in the test script instructions.

The 12 people you have lined up for this test need to be split into two distinct groups. One group will use solution A and the other will use solution B.

You should be looking to measure both quantitative (e.g. time on task and success rates, etc.) and qualitative metrics (how did they feel about this solution).

Ideally you want to disguise the true purpose of your test, e.g. you don't want participants knowing this otherwise it can pollute the data (participants can sometimes tell you want they think you want to hear, rather than telling you what they really think). So the construction of the test script needs to be carefully thought through. I typically give them a bigger goal to achieve, even though I might only be interested in one tiny portion of that bigger goal.

After the test is complete, you could swap the two groups over and repeat the test. The purpose of this swap is purely qualitative - you just want to see if the participants had a preference. This is not the primary purpose of the testing, but might reveal some interesting insights.

Answer 3

You're looking at A&B testing. Its a pretty standard method of usability testing.

Basically the key question to ask is- how would you put a quantitative value to usability testing one individual system? How do you measure whether a system has 'good' usability or not?

So doing things like timing how long it takes people to complete tasks (broken down into those they passed without help, passed with help, and failed), and the SUS survey, can help give you these numbers. Its then a case of comparing the two.

Ideally you should try and chose your sample well so that you've a very similar set of people testing A first and testing B first.

Comparing the same person on A then on B is also useful though obviously has the problematic variable of them being pre-introduced to the other system. It might also be worthwhile to have some people tested using A twice/using B twice-> learnability isn't so important for an onboarding system but this will help give you some data to show stake holders why the data for the same person using one then the other may not be the best.

Gathering qualitative feedback is important too of course. And there...it is largely down to your judgement.

Answer 4

2) With regards to the evaluation, your format can effectively employ the System Usability Scale. Just replace "the system" with "onboarding process"