I have a client that is very opinionated about what she likes and dislikes about my design and moves things around to where they "look right" for her. I want to test my original mock-ups vs her revised ones with users but they are only mock ups. I just want to make the point that users will be very confused if she hides some key features that set her apart from her competition - the opportunity to buy an item or get to the personalization process of her shopping experience.

Which would be most effective? 1) Do a simple test by asking users what they understand from the mock ups and where they would go to do what as open-ended questions; 2) Do "fake tasks" like asking "where would you go to...(desired action)?" Can I then do this with the different sets of mock ups or does it have to be one user per mock-up version?

I think this site has a lot of potential for success but am worried her apprehensiveness and micro-managing may be undermining the site goals. However, explaining this to her has been of no help. She thinks I insist on things because I am "passionate" not because I am just plain old experienced.

6 Answers 6


A/B testing is about optimisation. Mockups and user testing are about insight.

User testing (even with a small sample set) will help give you insights into what potential problem areas may be so that you can find ways of improving them. This should be your goal, not trying to optimise a design.

A/B testing, needs large sample sets to make any conclusion about which option is better, but gives you little insight into why that option is better.

Think of user testing as course refinement and A/B testing as fine refinement.

Given your situation, I would do some quick dirty usability testing to see if there are any clear problems with either design. It neither shows to be clearly worse than the other, then go with what your client wants.


For quick and efficient mockup testing, I recommend using any program that can link 2 images and Silverback for recording the users sessions.

  1. Build a small user panel (5-7 people) - more if you can...

  2. Write up a short scenario which tells your users what they should accomplish before testing the flows.

  3. Build 2 versions of the same flow using your mockups and hers. Mockups are linked using any app (InvisionApp, Power Point, Keynote, Omnigraffle, Axure, etc) in order to simulate an online experience.
  4. Use Silverback to record users' on-screen activity with sound and facial expressions.
  5. Once they're done, ask them which flow felt easier, more natural, etc.
  6. Summarize your users' feedback and discuss it with your client.

Your concerns are very valid. It's way too often that clients pretend to know what's best without realizing that they're not the primary audience. Good luck!

I'm not affiliated with any products mentioned above

  • Would you test both with the same users or, if you had larger panel, would you have different users? How many would you say you would need before you the results find their level? Commented Apr 13, 2013 at 20:54
  • The quickest solution would be to give the 2 flows to the same users and get their opinion. Now, what about going even further if you think its necessary : give half of your users the flow A first and then B. Give the other half B first and then A. Then compare their opinions. This way you'll filter the "first impression" as well.
    – SuperFluxx
    Commented Apr 14, 2013 at 14:53
  • As per the number of users -considering Cecilia's situation- I would test against around 10 and up to 15 users max, which would be quite time consuming already.
    – SuperFluxx
    Commented Apr 14, 2013 at 15:00
  • It sounds like your designs (or mockups) are overridden due to your client not having buy-in in the designs. You can do a simple think-out loud study with scripted tasks (and steps), even paper sketches or printouts might suffice. This process will still give you valid information to inform your design process.
  • Different sets of mock-ups will be useful if you are clear what each version will accomplish or how they differ from each other.
  • It is a common for clients to assume that they're the most typical user (sometimes they might be right) and in your case, there is definitely some disconnect between your client and the target user population.
  • It might help if you can bring in three to five users, give them a contextually valid task, and record a video/audio (or both) of the task flow with different mock ups. Better yet, have your client watch these sessions live (in a different room or remotely). With mockups, you don't need to have a long session, a ten to fifteen minute user interview might help your client to gain a different perspective. Also, make sure that your interview with users is not leading or biased and it might also help you to get a sign-off on the materials (questions, process, participant selection criteria,mockups, etc.) from your client before you bring in users. It might sound like some extra work to do this, but it will cut down the unproductive iterations for your designs.

Do an A/B test with OptimalWorkshop or a similar tool. http://www.optimalworkshop.com/chalkmark.htm

Take the two mockups and figure out what the task is. So yes, you want to do a fake task. For example, the task would be to add an item to the cart. You can do multiple tasks, some would be adding to cart, some would be to find more detail, you'll want to mix it up.

You want to compare apples to apples, so for example in task, "How would you add XYZ item to your cart", you'd want to do one with your mocks and another with her mocks. Visually they should look about the same. Mix it up with different types of scavenger hunt type tasks (eg. find details on xyz, or how do you view your cart). Once you've done about 15-20 users, you'll be able to find out how quickly users did it in the two sets of mocks.

You'll be able to objectively say, in set mockup set A, the user was able to find the checkout button on average of 3 seconds but in mockup B it took them 15 seconds with more errors.


A/B Testing is definitely the way to go... but without the resources to properly develop both suggestions, the results are likely to discarded by the highly opinionated individual.

The client has discounted your talent, experience and background. Unfortunately, they'll likely discount any testing, as well.


You've got a lot of good directions about what methodologies to use for testing your options. But, the one thing I’d like to call out even more boldly is that understanding what the success metrics are for your client. You won’t be able to adequately state what your testing uncovers if you and your client aren’t on the same page about what is most important for your designs. You’ve already mentioned one thing that is important, but are there others? Make sure the questions/tasks you propose will get you closer to good answers. Also, if there are any design decisions that can be supported by the research done on http://usability.gov, that can be another avenue for bigger data backing up decisions. (Note: They’re in the process of updating http://usability.gov this year and hopefully the research will be even more useful when it’s more current.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.