2

I'm looking to do a comparative test on a prototype with 2 variations. The prototype is an onboarding strategy for a product. 1 version is designed with persuasive cues (social proof techniques) while the other has no persuasive cues. The prototype is tailored to each participant, so the participant will see actual data from their social circles.

I'm looking for ideas on how to evaluate the prototype. I'm trying to identify if the onboarding experience with persuasive techniques is more likely to encourage users to participate in the product quicker, than the variation with no persuasive techniques.

  1. Are there any good models / tests for conducting this type of testing.
  2. What type of questions could be asked in order to effectively evaluate the test?

I will be testing with 12 participants, each participant will review both variations. I plan on randomly selecting which variation a participant views first in order to help eliminate any bias with being exposed to a particular variation first.

Thanks!

2

In a perfect world you could test your prototype with dozens of people separated in two groups that will test either one or the other prototype. With large enough numbers both groups will be somewhat consistent.

With just 12 test subjects, testing both prototypes you need to take into account prior knowledge.

You'll need to determine clear metrics. When is the test or task a success and how can I measure this?
Maybe speed is a metric. Number of clicks. Completeness of profile information. Number of social connections made. Whatever your app is about.

I can't give a decisive answer because I don't know much about your prototypes, but you could either give the test subject a task or just let them browse while thinking out loud (also important during test with task!).

Avoid tasks and questions that are suggestive and be wary when asking for opinions (don't ask which of the two they liked better, let your test results speak for themselves).

I hope this helps (a bit).

  • I'm prob going to do a significance test to see if the results are statistically significant. In order to do this I need to use a Likert scale in some form. So for example, I was thinking of asking users on a scale of 1 to 5 (strongly disagree to strongly agree) "This version is likely to encourage me to do x" and evaluating this for both prototypes. I agree with avoiding suggestive questions. I think with this approach I need to be careful of that. – UXG Dec 5 '16 at 16:13
  • I'd be surprised if your sample size was big enough to produce statistically significant results. – PhillipW Feb 4 '17 at 19:25
2

You have a proposition that you want to test - that persuasive cues are more likely to encourage users to participate in the product quicker, than the variation with no persuasive cues.

This is classic A/B test, and you need to create a test scenario which avoids introducing any bias, including your own, and you need to be mindful of leading the participants in the test script instructions.

The 12 people you have lined up for this test need to be split into two distinct groups. One group will use solution A and the other will use solution B.

You should be looking to measure both quantitative (e.g. time on task and success rates, etc.) and qualitative metrics (how did they feel about this solution).

Ideally you want to disguise the true purpose of your test, e.g. you don't want participants knowing this otherwise it can pollute the data (participants can sometimes tell you want they think you want to hear, rather than telling you what they really think). So the construction of the test script needs to be carefully thought through. I typically give them a bigger goal to achieve, even though I might only be interested in one tiny portion of that bigger goal.

After the test is complete, you could swap the two groups over and repeat the test. The purpose of this swap is purely qualitative - you just want to see if the participants had a preference. This is not the primary purpose of the testing, but might reveal some interesting insights.

  • Thanks for your suggestion, but unfortunately I'm unable to do an AB test due to resourcing etc, otherwise I would have. This method is far more suited to AB testing and being run over a larger population than 12 users. However, I' in a position where I need to evaluate it using an invision prototype so I'm thinking qualitative testing with some method of quantifying attitudes towards behavior in some way. So assessing how likely users are to be influenced or not, and why etc. Then user a likert scale to quantify their preference to one over another. – UXG Dec 5 '16 at 17:32
  • I do A/B testing with low number using paper prototypes as well as Axure prototypes. Why do you think it needs lots of users? You have 12 so split them into 2 groups of 6. You can still learn insights with low numbers. – SteveD Dec 5 '16 at 17:43
  • I am not sure you quite understand what A/B testing is. I am wondering if you are thinking it means building and deploying two different solutions to your whole user base, while you monitor a wide range of analytics? A/B testing can be done this way, but its definition is nothing more than comparing two designs with two groups of participants. Those designs can be paper prototypes, white board prototypes, Axure/Balsamiq/Invision/Photoshop prototypes, html prototypes or stubbed-out code prototype. 5 or 6 participants per design is OK for a usability test. – SteveD Dec 6 '16 at 9:45
  • For validity sake, you can't have the same participant do both versions, brings in a huge "previous knowledge" impact. Some sort of a split has to occur. – Petar Subotic Jan 5 '17 at 17:14
1

You're looking at A&B testing. Its a pretty standard method of usability testing.

Basically the key question to ask is- how would you put a quantitative value to usability testing one individual system? How do you measure whether a system has 'good' usability or not?

So doing things like timing how long it takes people to complete tasks (broken down into those they passed without help, passed with help, and failed), and the SUS survey, can help give you these numbers. Its then a case of comparing the two.

Ideally you should try and chose your sample well so that you've a very similar set of people testing A first and testing B first.

Comparing the same person on A then on B is also useful though obviously has the problematic variable of them being pre-introduced to the other system. It might also be worthwhile to have some people tested using A twice/using B twice-> learnability isn't so important for an onboarding system but this will help give you some data to show stake holders why the data for the same person using one then the other may not be the best.

Gathering qualitative feedback is important too of course. And there...it is largely down to your judgement.

  • 1
    I don't think A/B testing would be quite suited. The test itself involves the user clicking a login screen and going to a main page. On the main page a popup will display showing persuasive cues to encourage the user to perform a specific action. (the other prototype will be the same, without the cues) The prototype will be done with Invision, as opposed to being coded and proper split testing ran in the product. So the test is more about identifying from users if persuasive cues are likely to have an effect on them participating at a quicker rate, from seeing both variations. – UXG Dec 5 '16 at 16:07
  • There are no tasks as such I'm looking to measure (time on task etc). They are more reacting to information as opposed to figuring out how to interact with form elements, components etc. – UXG Dec 5 '16 at 16:08
  • I think you should go with A/B testing itself, since Usability Testing is mainly used to check the usability rather than preference. – Saurabh Kumar Suman Dec 5 '16 at 16:24
  • Unfortunately A/B testing is not an option due to development resources etc. So I need to be able to evaluate with a comparative test in some way as opposed to a usability test. – UXG Dec 5 '16 at 16:28
1

2) With regards to the evaluation, your format can effectively employ the System Usability Scale. Just replace "the system" with "onboarding process"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.