What is the optimal way of structuring a multiple choice preference evaluation?

Question

Say I have a set of photographs and I want to ascertain the most popular one based on a group of people choosing their favourite.

What is the most optimal way of structuring a survey to derive a confident estimate of the most preferred photo with the smallest sample size.

For example would it be better to do...

• Show all photos and ask people to pick their favourite
• Show a subset of photos and ask people to pick favourite
• Some other structure

Does the optimal structure change if you have more options?

Answer 1

You want something like comparative judgement, which is an excellent way to rank things in order.

Basically you take two of the images and say 'this one is better than that one'. Then ask again, for a different two images. You can arrive at a solid grading in a very short space of time, with a pretty small survey sample size.

To get more reliable results, you pick the ones which sometimes get upvoted, sometimes get downvoted, and display them more frequently in the pairs.

Answer 2

If there are tons of images, then showning all the images at once doesnt really help and confuses the user. The photographs can be categorized(abstract, portraits,etc) based on the type of photographs you want to show, and sequentially make the user go through categories based on the users preference, or its totally your call. This is something i have in mind.

If there is no category as such, then showing a small set of photographs is good, this will not confuse the user. However the time taken in this, is much more than showing the images to the user all at once.

PS: the demo in the link is just an idea on how you can showcase your photographs, and also if one image is marked as favorite in "all" category, make sure its highlighted in the other category as well.

Answer 3

Consider these questions (probably you can ask yourself)... What is your plan to do this exercise? How would you choose the users? Are you planning to do it online or offline? Do you present these pictures online and ask the users to pick their choice or Do you take hard copies and show the people to choose from them? or Are you planning to have some interviews with users / people?

What I am thinking is...

1. Identify the target uses
2. If there are lots of images, I'll try to shortlist the images to a few numbers. But, I prefer to involve few of my colleagues / friends to do it, rather than doing myself and reward them (maybe a pizza or a coffee or whatever for helping me).
3. I'll do the same exercise with target users to refine the selection from the list which was chosen by my colleagues / friends.

Doing this exercise with the target audience is a key here, otherwise, the whole research might not give you full result what you are looking for.

I am not sure what type of photographs you are trying to shortlist, but it completely depends on the target users / audience. Because, the age / gender plays a major role here.

Answer 4

You are trying to get subjective data based on small sample size. This is not likely to yield great results. Like other answers have mentioned you need to identify target users and categorize images if they are many.

Also take into account the Primacy and Recency bias. Try to randomize your image sequence for users to get more balanced data.

Most importantly, identify the parameters you are are evaluating these images. A small sample size based popular ranking is likely to fail because each has a very personal idea of why something is their favorite.

Hence it is very important to first come to a set of parameters on which you are evaluating the image set. Otherwise, the data gathered will be very random and can not be used objectively. If you finalize such parameters, you can then ask people to grade the images on a scale or rank the images relatively for each parameter. Considering your business goal you can weigh the parameters to reach the result.