In usability testing, what consensus from users is enough to throw out an idea in future iterations?

Question

I am in the midst of adding about 100 images to one of my current projects after completing a round of usability testing on a low/medium fidelity prototype yesterday. I included ideas for 4 pages in the test. Finding and editing the images for this next round of testing could take a week or two and involve a good deal of back-and-forth with the client.

Page #1: 80% of users preferred version A over version B.

Page #2: Users saw two ideas. 55% of users preferred one of them; 45%, the other.

Page #3 showed users four ideas, and one of them was preferred by 80% of users.

Page #4 showed users three ideas, with some support for all three and no ideas getting a majority.

I'm doing this project as a volunteer for my portfolio. So saving time is important, but so is launching the sites on schedule, getting good press for this redesign, and landing future projects with paying clients. I'm planning two more iterations of usability testing after I get the site to beta (it is a WordPress site).

According to best practices, do I need to continue developing each of these ideas and show each idea again in the next round of testing? Should I throw out each idea that didn't win in this round of testing? Or which ideas should I continue to develop?

Answer 1

The ultimate answer to your question will depend on the number and quality (were they the right type) of users with whom you did the testing.

I would definitely think that 80% is good enough.

When it's almost 50/50 or where the preference is not obvious:

1. You can analyze to see if there are patterns in the differences between users and their preferences (e.g. do female users prefer one image and male the other? does it come down to nationality or age?)- if there is a pattern, try to identify the user who represents your target user best and go with that.
2. If you tested with a larger number of users (I would say 10+ is not bad), it may well be that all solutions are viable and would work - so just pick one (images are subjective).
3. If no pattern and smaller number of testers, if budget allows - consider testing again, but perhaps at later stage as part of testing other features (so that its less wasteful).

Good luck.

Answer 2

What you are referring to here is called UAT (User acceptance testing) and not usability testing.

That said, The truth of the matter is you cant satisfy everyone and there will always be users who dont agree with what you have proposed and will have a different view point. It is up to you to take the call on what design decisions to take and how much influence those dissenting stakeholders hold and what is the most important to ensure the design remains usable and consistent to the proposed branding guidelines

That said, I am afraid I cant give you a definitive answer on what to do next but here is what I would do

Page #1: 80% of users preferred version A over version B. - Take the 20 % who didnt like the version A and ask them what features they preferred in version B. Analyze and see if those features can be accommodated but dont affect the overall design (also determine how significant those 20 % are and what how a design change or update can affect your over-all design)

Page #2: Users saw two ideas. 55% of users preferred one of them; 45%, the other. : Similar approach as above except this time you will have to ensure you accomodate for the people who are the minority since they are a minority by a small amount. You might have to come up with an updated design with the conglomeration of features for that page and see how it gels with the group in another round of quick and dirty testing.

Page #3 showed users four ideas, and one of them was preferred by 80% of users : I think you should be good here unless the differing user group is a key stakeholder whose opinion you have to choose.

Page #4 showed users three ideas, with some support for all three and no ideas getting a majority. : This is going to be your biggest challenge. I recommend doing an information architecture session with a kind of modified card sort (the card sort will contain details of what goes on that page) and try and get consensus on what your page should contain

Note: This would work if you are trying to get consensus on the layout but if you are having issues with the colors, you will have to take the call with regards to the branding guidelines of the client you are working for and design principles to ensure the user can easily assimilate the content as needed.