# How to measure the difficulty of a question

This might be a more different question about difficulty than others, and might be a hard one to answer.

I am designing an educative application for tablets. The concept consists of a game where kids aged from about 10 to 12 years oldlearn the proces a certain product or service undergoes. Like the path from a cotton plant to the t-shirt they wear.

I have done my first UX test and had some useful responses. However I am not sure how to measure the difficulty of a question. Is it too hard or is it too easy? I made notes, did observation. Asking questions about this was too suggestive in my opinion.

Have you had this problem before?

Greetings,

Emnaon

-
Can you collect how often the question is answered right vs wrong? – Luke Jan 6 '14 at 20:52
It's more pedagogical field, still you could dig into Item-response theory, which claims among other "For a given ability level, the probability of a correct answer increases as item difficulty decreases." So applying some statistics to the collected data, you could infer the question difficultiness with some probability. – Alexey Kolchenko Jan 6 '14 at 20:55
Perhaps Cognitive Sciences could help. – Marjan Venema Jan 7 '14 at 8:06
Item-response theory and test development are the place to find more information. In the U.S., ETS is a test development organization. This is a link to a page on their site explaining test and test item development. Click on the Watch Now link (next to the big Q) for a video about evaluating questions in a test. – user1757436 Jan 8 '14 at 15:53

Difficulty of anything is subjective data and will vary from person to person. To identify difficulty of an application we have no other means but to undergo user-testing. To identify difficulty of a question, you would not just need to test it but test it with fairly large group of participants to be sure that what you calculated was difficulty of the question and not the efficiency of participants. For a class of students, performing bad on the question may mean they didn't learn the topic well instead of question being hard. So you will need to test same question with multiple groups of students who had their learning sessions different from the other and only then you can identify questions which are persistently hard and multiple groups were failing on the same.

Apart from this required user-testing, for text-book questions, we can have additional matrices which will help you estimate their difficulty to some level.

• If a question requires understanding of multiple topics, you can presume it would be relatively hard to answer than the one which requires understanding of one topic only.

• If a question has similar example given in the learning content, that question would be relatively easy solve than a completely new scenario.

• Questions having multiple parts would be difficult than the single part question.

• Questions in which result of a previous calculation is carried forward would be more difficult than the ones which have independent sections. It may not be called difficult but the percentage of candidates calculating correct result would decrease in such case.

• Questions which require recalling of a number or formula would be more difficult.

• In case of multiple choice, proximity of possible answers with each other makes the question hard. More closely related answers you have, the more difficult answer would be.

-