I asked an open-ended question in a survey about one of our app's features (That is somewhat similar to its search feature) to find out why users would use this instead of search. I've got a large number of responses (1000+).
My question is what's a good approach to analyzing the responses and derive insights without having to go through each response. Right now I'm searching for the most frequently used keywords and grouping them. is there another more effective way?
Text Mining
Some researchers have calculated word frequencies, but raw frequencies by themselves are not very reliable or informative. For example, if 40% of your users used the word “helpful,” you might feel pretty good about your feature, not realizing that most of the time they’re saying “not helpful.”
Modern text mining or content analysis uses sophisticated algorithms (often qualifying as AI) to look for semantic patterns of words. But even with such computational horsepower, there is still substantial human skill and interpretation in deriving insights. There’s various books and tools out there for this, if you google for them.
Alternative: Manually Categorize Responses
But, frankly, unless you already have expertise in text mining, it’s probably not worth doing for most usability questions. Usually, you just want to get a reasonable impression of what are people saying. So, use the semantic pattern-recognizer you already know how to use: your own brain.
Systematically but manually reading and categorizing answers will take less time than learning what you need to learn about text-mining, even if you read all 1000 answers, and, as I say below, you don't necessarily need to read them all.
Data-driven Category Definition
Derive categories for classifying the answers as follows:
Take a random sample of about ten answers, and read them to see what categories of answers they naturally seem to fall into. For example, maybe four were along the lines of “I used Feature A because I didn’t know there was a Search.” Another three might be, something like, “I tried search, and it didn’t work for me.” The remaining three may seem to each be in categories of their own. Sort the answers into these 5 piles (three being a “pile” of one answer).
Take another random sample of 20 answers and see how they fit into these categories. Refine the categories as you go: were two separate categories really the same? Is one category really two separate ones? Starting writing down rules for deciding where an answer goes in one category or another. Keep the categories broad and general –avoid having rules that depend too much on the nuances of the wording. People just don’t write that precisely, so you’d just amplify noise if you base categories on distinguishing, say, “I didn’t know there was a Search,” and “I didn’t think there be Search.” The point is, the user thought this was their only option. You may find some answers fit in more than one category, and that may be fine. If the user is saying, “I used Feature X because of Reason 1 and Reason 2,” then you want to capture that.
Do another sample of 20 to check if your rules for the categories are working. Check back on your original 10 answers and make sure you’re categorizing them consistently with rules as they have evolved. Try to arrive at a set of categories where only 10% of answer fall into “miscellaneous.” This doesn’t include answers that are just plain uninformative, like “I use Feature X because I wanted to.” You can still categorize and sub-categorize uninformative answers.
Improving Objectivity
For less subjective categories, work with colleagues. One can do the same steps above in parallel with you, using separate samples, and then you can meet to reconcile differences in your categories or rules. Once you are satisfied with your written rules, give them to another person, and have them categorize the same 50 you already categorized. Ideally, s/he should classify the answers in the same category 90% of the time (80% may be acceptable) without any assistance from you, other than handing over the rules. Be prepared for another round of refining the written rules and trying again.
On the other hand, if you and your client are satisfied with you saying, “I read the answers, and my impression is users are predominantly saying A, B, and C,” then such academic-grade cross-checking may not be necessary.
Complete the Analysis
Once you’re only making fine adjustments to the categorization rules, then you’re ready to go with categorizing the rest of the answers. With clearly written rules, you can go through the answers to single question quite quickly. If, as typical, the users wrote only a sentence or phrase for each answer, you can categorize each in a few seconds. You can go through a 1000 in an hour or two.
Analysis is predominantly the percent of responses in each category. You can also correlate the categories with other non-open-ended variables you collected (e.g., do long-time users have different distributions of answers than noobs?).
Interpretation Limitations
Be prepared that what you might discover is that you didn’t ask quite the right open-ended question. Your users may be interpreting it in a way that’s not useful. If you asked, “Why did you use Feature X?” you might get mostly answers like, “To find a Y,” which doesn’t really tell you anything the analytics won’t already tell you.
You cannot expect too much deep introspection from your users. They were busy doing the task, not observing themselves and mentally noting why they made every move and not some alternative. Expect superficial answers, and trust only superficial answers. And be aware that users may simply not notice or remember why they did something many minutes after they did it. You might find you need to do a usability test, where you can sit by the users while they think aloud, and you can ask them why they did something the moment they do it.
Do you need read all 1000?
But suppose your users typically wrote entire essays (how helpful) rather than just a sentence or two. Or suppose you have a bunch of open-ended questions. Do you need to work late tonight?
If all you care about is determining the top few categories of answers, then no. Increasing sample size only increases the confidence you have in the percent that each category represents. With random sampling, the precision of those per cents depends on the raw number of answers, not on the proportion of the total sample of 1000. If you only care about getting the per cents to within about +/-10%, then a random sample of 100 is enough. That means that by the time you’ve made your categorization rules, you’re already about halfway done.
On the other hand, if it’s important to determine with high confidence that Category A beat Category B, even though they’re only 4% apart, then, yes, you need to read all 1000.
Need More Help?
Googling simply “analysis of open-ended survey questions,” will yield helpful links.
Here are two use-cases that use R / Python for your task:
