So I have a series of concepts for a redesign of part of the product I work on. 4 concepts total. I want to test each one to decide on an overall direction for the redesign so that it's based on user feedback and not anyone's opinion or gut. But I'm not sure the best way to write the test so that I'm getting helpful feedback. The more I think about it, the more it seems like I can't just have 4 separate groups use one concept and give feedback on that one concept. I don't feel like I'll get anything definitive from that because they have nothing to compare it to.

Some other test set-up ideas I've had:

I could have one larger group use each of the concepts and choose which one they liked best through survey style feedback.

Or I could have 4 groups try the current design and then one redesign concept and give feedback.

Has anyone tested this way before? Any insight into what got you the best feedback?

2 Answers 2


This is the standard issue with any kind of A/B testing. The issue with testing more than one design with a single person is that people will always have a bias for the first thing they are shown and get to use. So if you are exposing more than one design to your user base then you will have to also interchange what design is shown first.

But if you want to run with the 4 groups then the best approach would be an identical set of tasks for each design and a graded approach to capturing feedback. What I mean by this is that you identify the idea path to complete each task within the interface and then grade the user on if they completed each step in the ideal path. Buy assigning simple values of:

  • 3 (complete without issue)
  • 2 (required minor help)
  • 1 (required direct guidance)
  • 0 (failed to complete)

you will get results for each task as a whole that can be compared across designs. You will also get measurable feedback on where within a task a given design breaks down. Take opinion out of it and look at a designs ability to facilitate successful completion.

You could also capture feelings and emotions afterwards to get their gut reactions to the design as well since this does impact how the user approaches what they see.

  • Thanks for the response! If I'm only able to use remote unmoderated testing right now, is there still a way to grade the path completion in a non-binary way? I'm not going to be able to give guidance should it be necessary. Maybe like 2 - complete without issue, 1 - complete with some issue, or 0 failed to complete? Should I maybe look at other things like time on task to determine difference in the design strength?
    – amei
    Aug 30, 2016 at 17:52
  • Attempt this sound testing method. you will be surprised to see how helpful it can be? Aug 30, 2016 at 18:11

Actually I think remote, unmoderated testing will provide you with potentially more feedback in a much shorter time than if you try to run any moderated groups with that many concepts. Depending on the tool you use, your study design, and good recruitment, the upside is that you can get many more people responding to your designs, which is a good thing. The downside is that with unmoderated you need to design and pilot your study carefully to make sure it is getting what you want out of it, that the prompts are clear, and when people report completion you can tell whether or not it was full completion, partial, failure, etc.

I think that having a large number of people each responding to a single design will, in aggregate, give you a better overall picture than one group trying to choose between all four.

However if you have the time and resources, why not do both? Run a large study where people are randomly given one of the four designs, split so you get an even mix, and at the same time run a single study where each participant is shown all four (preferably in random order). I think you will get very different results there, since each design is likely to affect the way they interact with the next.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.