There's a particular feature of our (B2B) app that seems to be causing a lot of confusion. (It involves lists of urls, some of which can be quite long, and giving different statuses to different ones. On the back end, there's a bunch of boolean math involved, but obviously, we're trying to hide that from the users.)

Part of the confusion is that the domain is complex. We've come up with four designs to try to clarify things, and I want to test them. Any advice on how to keep the testing manageable? A few of challenges:

  • we don't have enough users to do automated A/B testing of any kind
  • it's a somewhat complex conceptual model, which probably requires at least a little domain expertise to understand
  • despite the above, our users aren't necessarily very technical

2 Answers 2


It is possible to do a controlled (lab) test using a within-subjects design. For this, each participant would be exposed to each condition (design). There could be order effects (i.e., they learn as they go on) so counterbalance the order in which participants work through each design (so participant 1 gets designs 1,2,3,4; participant 2 gets 2,3,4,1; participant 3 gets design 3,4,1,2; and so on). This helps prevent order effects.

The difficult bit is working out what you want to measure. That depends upon the goals that the software is supposed to achieve.


I'd suggest using something like Silverback (not sure if you have Macs or not, but it's just an example of an app that does what I'm thinking of) to actually set those-who-will-be-end-users specific tasks within the various designs and see how they perform.

It'll let you see if they can work out the steps required within their domain; you can see where the trouble spots are and maybe adjust your designs based on that. Plus it'll give you more than actually asking the user to report back. They may not accurately report their issues or be able to articulate them in the same detail as you can get by watching them in action.

I've only done superficial guerilla testing in the past, but it's a simple idea and seems to work well when you've limited resources.

  • Silverback looks great, thanks for that link! I guess I was hoping for specific advice related to the challenges of testing many design variations, though.... :)
    – sprugman
    Jul 7, 2011 at 19:16
  • Oops. Sorry. I didn't quite pick that up. One of the biggest problems with many variations is actually getting agreement. In my experience, different users have adapted their way of working to cope with the systems they know, and when you show them something new, they're trying to shoehorn their existing methods into your design. It rarely works. It's not always easy to get consensus.
    – Sam K
    Jul 8, 2011 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.