What do users expect from a multiple answer question?

Question

I have several testing/survey systems. One of the big debates is how to deal with questions that require multiple answers.

Configuration #1

The user must have the exact check boxes marked. If they do they get "1" point, if they don't they get "0" points.

Configuration #2

The user would get points for each right answer they choose. If there were two right answers out of four then each would be worth ".5" points. And then the question author could give negative points to the other answers (you can never score worse than a "0" on a question). So if the other two answers were "-.5" and the user selected the two right answers and one wrong answer they would get ".5" points out of 1 for the question.

Configuration #3

Same as #2 except that one wrong answer means you get a "0". So they would retain half credit if they only marked 1 of the 2 correct choices.

What is the most accurate way to track scores? And what do users expect when answering multiple answers? From my standpoint the whole reason to use multiple answers is to create a harder question.

Answer 1

This is an interesting question, and I think you should also consider the answer from a learning/instructional design perspective rather than just the user's perspective.

Configuration #1 suggests that there is only right and wrong answers and nothing in between, therefore the questions will need to match this philosophy. I think it applies quite well when perhaps the details are quite important and you can't get something 'almost right', such as is the case for safety related information.

Configuration #3 suggests that there is an optimal answer, but it is not necessary to get everything right in a valid result, so the questions are ideally structured to try and differentiate between people who cover things in more depth and knowledge versus those who only cover the broad overview of things.

Configuration #2 is interesting as it assigns negative scores for incorrect answers, and so it is most difficult in the sense that you can get the lowest scores out of all the configurations and hence the most powerful in segmenting the students of different abilities.

Whilst I don't think there is an optimal configuration, the emphasis should be that the type of scoring matches with the intent of the question. So in fact I would probably even advocate for a mixed configuration set up if the purpose is really to ensure that students reach their learning objectives rather than trying to separate the students very clearly according to their learning abilities.

In fact, there is also an interesting take on whether you reveal the scoring configuration for each question, as students will also opt for different strategies for answering the question based on their own confidence about getting the answer correct.

Answer 2

From my standpoint the whole reason to use multiple answers is to create a harder question.

Open answers are the hardest question types since they need more precision about the answers. However, evaluation of these questions are hard for machines.

if you have to bind multiple selection type of questions and if your aim is to make the question harder, all or nothing strategy will be suitable for your solution.

In addition to this, reward and punishment strategy can also work however too much negative points can also have negative effects on motivation to continue the rest of questions or creating unwanted behavior like restarting the test.

Answer 3

Configuration 2

Configuration 1 and 3 will fail!!

Reasons:

1. Consider if I tick all the answers in the multi-choice question, I will get the full marks in case of Configuration 1 and Configuration 3. So they will fail!!

   2. Only configuration 2 is perfect as the user will tick only if he knows for sure.