Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I am in the midst of adding about 100 images to one of my current projects after completing a round of usability testing on a low/medium fidelity prototype yesterday. I included ideas for 4 pages in the test. Finding and editing the images for this next round of testing could take a week or two and involve a good deal of back-and-forth with the client.

Page #1: 80% of users preferred version A over version B.

Page #2: Users saw two ideas. 55% of users preferred one of them; 45%, the other.

Page #3 showed users four ideas, and one of them was preferred by 80% of users.

Page #4 showed users three ideas, with some support for all three and no ideas getting a majority.

I'm doing this project as a volunteer for my portfolio. So saving time is important, but so is launching the sites on schedule, getting good press for this redesign, and landing future projects with paying clients. I'm planning two more iterations of usability testing after I get the site to beta (it is a WordPress site).

According to best practices, do I need to continue developing each of these ideas and show each idea again in the next round of testing? Should I throw out each idea that didn't win in this round of testing? Or which ideas should I continue to develop?

share|improve this question
add comment

2 Answers

up vote 3 down vote accepted

The ultimate answer to your question will depend on the number and quality (were they the right type) of users with whom you did the testing.

I would definitely think that 80% is good enough.

When it's almost 50/50 or where the preference is not obvious:

  1. You can analyze to see if there are patterns in the differences between users and their preferences (e.g. do female users prefer one image and male the other? does it come down to nationality or age?)- if there is a pattern, try to identify the user who represents your target user best and go with that.
  2. If you tested with a larger number of users (I would say 10+ is not bad), it may well be that all solutions are viable and would work - so just pick one (images are subjective).
  3. If no pattern and smaller number of testers, if budget allows - consider testing again, but perhaps at later stage as part of testing other features (so that its less wasteful).

Good luck.

share|improve this answer
Hi! Welcome to StackUX! Please check out the faq and about to learn more about the Stack Exchange Community. Could you provide some references to back up your recommendations? –  norabora May 14 '13 at 21:05
Hi - thank you! My answer comes from experience and from knowledge gained by reading books and few visits to Usability Week –  Laura Papla Ford May 14 '13 at 21:21
@LauraPaplaFord Thanks! I moderated tests with 5 users and had 10 more evaluate it via a survey. I was trying to test with 10 users, but the client has a low budget and the rewards we were giving (so far) were not enough to recruit more testers, hence the survey. –  David May 14 '13 at 23:58
add comment

What you are referring to here is called UAT (User acceptance testing) and not usability testing.

That said, The truth of the matter is you cant satisfy everyone and there will always be users who dont agree with what you have proposed and will have a different view point. It is up to you to take the call on what design decisions to take and how much influence those dissenting stakeholders hold and what is the most important to ensure the design remains usable and consistent to the proposed branding guidelines

That said, I am afraid I cant give you a definitive answer on what to do next but here is what I would do

Page #1: 80% of users preferred version A over version B. - Take the 20 % who didnt like the version A and ask them what features they preferred in version B. Analyze and see if those features can be accommodated but dont affect the overall design (also determine how significant those 20 % are and what how a design change or update can affect your over-all design)

Page #2: Users saw two ideas. 55% of users preferred one of them; 45%, the other. : Similar approach as above except this time you will have to ensure you accomodate for the people who are the minority since they are a minority by a small amount. You might have to come up with an updated design with the conglomeration of features for that page and see how it gels with the group in another round of quick and dirty testing.

Page #3 showed users four ideas, and one of them was preferred by 80% of users : I think you should be good here unless the differing user group is a key stakeholder whose opinion you have to choose.

Page #4 showed users three ideas, with some support for all three and no ideas getting a majority. : This is going to be your biggest challenge. I recommend doing an information architecture session with a kind of modified card sort (the card sort will contain details of what goes on that page) and try and get consensus on what your page should contain

Note: This would work if you are trying to get consensus on the layout but if you are having issues with the colors, you will have to take the call with regards to the branding guidelines of the client you are working for and design principles to ensure the user can easily assimilate the content as needed.

share|improve this answer
Sorry, but this isn't user acceptance testing either. Not even according to the link you provided. User acceptance testing has to do with (representatives of) the client accepting the developed product in its entirety and signing off on your responsibility to deliver that product. I think you might mean a/b testing? To quote from the linked article: "UAT, or User Acceptance Testing, is a term used in software engineering to describe when the client would give final approval for the built system to be delivered." –  Marjan Venema May 14 '13 at 20:23
@MarjanVenema I was thinking of mentioning A/B testing in the question - that's how I think of it too - but went with usability testing because it seems more generic to me. From what I've read today about A/B testing, it seems that is for an at least partially developed product. Mine hasn't gone into formal development yet; it's still in Axure. The way I understand UAT from my former jobs in software development, UAT takes place after development and integration testing are done, and I'm not there yet. –  David May 14 '13 at 23:50
@Mervin Thanks. The conglomeration of features for page #2 seems like a good approach for this. For page #4, all 3 ideas contain the same information but present it with different layouts. Should I throw out the Page #4 idea that only got 20% of the vote, so to speak - and keep fleshing out or combining the top two ideas? The client has some restrictions regarding color schemes but gives me some latitude. This test was mostly a test of layouts, findability, and interactions such as signup forms - plus the "which version of the page do you like better?" questions. –  David May 15 '13 at 1:11
@David: yes, exactly right. UAT indeed takes place after development and integration testing. –  Marjan Venema May 15 '13 at 5:57
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.