Asking users to guess a distribution of answers, rather than a single answer

Question

So, I want to make a (semi serious) quiz that asks users to guess what most other people think on a range of controversial topics. At the end of the quiz, users will be informed of where they were most wrong about the views of others.

(I think there used to be a quiz show on telly that did this sort of thing - Family Fortunes perhaps).

I already have survey data in which most questions are answered on a five point scale:

agree strongly
somewhat agree
neither agree nor disagree
somewhat disagree
disagree strongly

as well as

'depends' (which I may choose to discard)
'refused to answer' (which I will discard)

Now the problem is that for some questions, the distribution of answers will have a wide spread (e.g. from agree strongly to neutral), or even be twin-peaked (for example some topics may polarize people into strong agreement/disagreement, or a lot may answer 'depends'). I don't want to score down a user for picking the 'wrong' popular response when several responses are near enough equally popular. But I think drawing a histogram is too much to ask of the statistical skills/level of engagement of the average user.

So how best to ask the user for their opinion on the distribution of answers to each question? Bearing in mind my aim is to determine the questions for which they most/least "understand other people" (in this limited sense!).

EDIT

Jayfang's answer is quite good. However, I'm starting to think a simplified histogram is appropriate, after a lot of combining categories and eliminating minority responses first. The issue then is how to present a histogram concept in simple terms (which will be renormalized to sum to 100%). I'm thinking a series of buttons from 'less popular' to 'more popular' by each response, with a footnote that if you click the same for all of them (e.g. all less, or all more) that is just saying they are all the same.

Answer 1

One way is to pre-bucket the interesting responses for each question, and then get 'users' to guess which "bucket" goes where.

i.e. Two answers below each got roughly 40% of the votes, which two answers do you think got the votes?

1. agree strongly
2. neither agree nor disagree
3. somewhat disagree
4. disagree strongly
5. depends

For a flat response question can only ask for the exception(s) i.e. the one bucket without any votes.

An alternative for a flat response is to allow an additional "None" choice. i.e. Which response got 50% of the vote?

... 6. none

Answer 2

You can describe the most common distributions in words and then ask them to take a guess at the parameter of the distribution. For example:

"All household cats should be spayed or neutered". What is your estimation about the public opinion on this question?

a. Most people have the same opinion, and it is

i. Agree

ii. Neutral

iii. Disagree

b. There are two camps, one with people who are all for it and one with people who are fully against

c. There is no strong tendency, roughly an equal number of people holds each of the possible opinions.

You say that your target population has so little prior knowledge in statistics/psychometrics that they would be overwhelmed by a histogram. This means that they probably don't attend much to problems of the kind you are asking, and don't have a well articulated opinion on the subject. So, you can't get a guess more precise than ones the people have already encountered and noticed in everyday life.

Maybe the precise distribution of public opinion on cat neutering on a 7-point Likert is bimodal with one peak twice as large as the other one and also skewed to the left. But with a target population like yours, nobody will make that guess. The precision of the representation of the public opinion seems to be low; so it is entirely permissible to measure it with an instrument of low precision (like the one I proposed above, or Jayfang's).

But if you still think that people guessing the correct histogram out of a lineup will add value to your research rather than the above description, you can use a piechart.

Piecharts don't enjoy much love from data scientists, but they have one large advantage in your case: The average person out there knows how to read them. And having very few segments (5 to 7 for a typical Likert), you avoid one of its largest problems, the overcrowding of the circle with a myriad of unrecognizably tiny segments. You will have to do a pilot study of course (should be doing one anyway, no matter what instrument you choose), but my estimation is that the average person will be able to work with it.