Is there a quantitative measure I can use in my results for an open ended question in user testing?

Question

Is there a quantitative measure I can use in my results for an open ended question in user testing? For example, if I ask the participant during the testing "Can you tell me, in your own words what this page is trying to tell you." the success will be them correctly explaining the info in the page. But when I come to present this information to other stakeholders, is there a way to represent the responses in a quantitative way?

The only measure I can think of is 3 out of 5 participants correctly described the info on the page.

My other sections in the user testing are marked out of a scale of 0 to 3. With 0 being unsuccessful, 1-2 needing a degree of help, and 3 being successful. Can I use a similar system for the above open ended question?

Answer 1

From my perspective the Likert scale of measure that is 0 to 3 rating can be measured quatitatively however for the open-ended questions you cannot present it in a quantitatively as the user's description is not precise as is measured on the Likert scale. They are subjective in the sense, for example, you cannot measure the word 'somewhat' quantitatively in the open-ended question. Beacuse it is a subjective response and can be interpreted in various ways. The open-ended questions will have to analysed qualitatively and presented as "Out of 5 users, majority of them, around 70 % favored 'Somewhat' agreeable as compared to 30% who informed Unfavorable'.

That's my take on the problem.

Answer 2

There is a well-established way if you use the means-end-chain laddering technique. This semi-quantitative research method allows researchers to ask open-ended questions, but the findings could be analysed and illustrated in a consumer decision map or hierarchical value map (HVM) [see an example here].

This latest open-access article published in the highly-ranked Journal of Retailing offers excellent details on how the laddering technique could be used. First, read details under Phase 1, i.e., Development of a hierarchical value map (HVM) using LadderUX, in the article. At the same time, review Figure W1.1 and Table W1.7 in the Web Appendix of the article to see how the findings of the laddering interview can be presented in a quantitative way. The HVM presented in Figure W1.1 can be developed using the easy to use online tool LadderUX.