Generative research - methods for finding areas of improvement in an existing app

Question

I'm frankly surprised by the lack of questions on here about generative research. This is the tenth question ever posted here about it.

Regardless, I'm finding myself at an impasse in approaching research for businesses who just ask me point blank "Find problems and make it better." Surveys can be helpful but they're better when I have some basis upon which to write the questions, otherwise I may just end up validating my assumptions. I don't like the idea of doing my own heuristic analysis because again, I'm just testing my own personal biases and not inquiring on behalf of the user. Most businesses don't like the idea of having to shell out the money for lab tests/contextual inquiries right away because they feel there's no guarantee in their mind that I'll identify valid areas of concern based off of 5 or so user interviews. Which is more a testament to my ability to communicate the value of UX rather than the method.

This has been a sticking point for me and I'm ready to fix my problem. What are better ways to not only conduct generative research to identify areas of concern that involve the user, and is generally going to be well received by the product owner? Thanks.

I suppose with existing products it may be an idea to just look into evaluative research methods but I would like to include generative as well.

Answer 1

Ethnography would be ideal, if you have the opportunity for it. And just document/record everything you can while in the field... ideally, with them using the product or prototype. This doesn't have to be expensive and might actually be more worthwhile than building a lab. Diary studies can also be effective ... maybe incorporate an incentive like "complete this diary study and get a $20 gift card" or something like that. Seems tacky, but can be effective... go lower than $20 if price is an issue. I once got a 61% sort completion rate (33/54) on a card sort study, largely (I bet) by incorporating a $20 gift card incentive.

Answer 2

Do you have the option to constantly ask questions to the client? If you can start by understanding the entire scope of their business and break it down into stages or smaller chunks, you can start focusing on each of them with more and more specific questions.

When you reach the point where more information is needed and the client does not provide, make the case to gain access to the users to fill up the holes in the research. Ask the client what you would ask the users, if they cannot answer, tell them this is why access to users is important.

In the other hand, you could take manners into your own hands and seek the users out to prove your point to the client.

Finally, if nothing works, reconsider if you want to spend time working for clients who pay but do not value your input.

Answer 3

Surveys and interviews rely on self-reported data, which UX researchers tend to trust less than direct observation. It's less accurate since people tend to forget important details and

In contrast, usability tests let you see first-hand what people do and where they have troubles. A think-aloud protocol gives even better results. You'll see results after just 5-6 of these and, if you're given no budget, you can do them with coworkers in (for example) Accounting who don't use the system. Even people outside of your specific user types can come across usability problems. No need for fine-tuned recruiting. One advantage of doing this is that you can use a screen recorder to gather visual and audio evidence that's often pretty persuasive with skeptics.

Heuristic Evaluations use objective principles that aren't that hard to apply without introducing your personal biases. (Have three or four coworkers do the HE if it's a problem for you.) This is a very useful - and cheap - research method and I'd encourage you to find a way to do it that you're comfortable with.

Good luck!

Answer 4

I think interviews are the right tool for this job so long as they’re conducted purposefully. So have just enough of a topic in mind that you recruit the appropriate interview subjects, but don’t over-script it or impose parameters on what it is you’re looking for. Consider all the data as relevant until proven otherwise.

Separate activities of data collection from data interpretation and trying to look for actionable patterns.

It’s a few years old but Steve Portigal’s interview with Gregg Bernstein provides what I feel is a great practical example of how to research people to figure out what you should work on.

“...the idea is you let the data dictate what’s important. You don’t put parameters on what you’re looking for. For instance, one of the ways we collect information from customers is through email. They can give us feedback about our app.

“This isn’t tech support, and it’s not a survey so much as just, ‘Send us an email about something you have to say about our app.’ That is just a data dump. People will give us ideas or they’ll talk about pain points that they have, which aren’t really technical issues. It’s just how our app could make their lives a little easier if we changed one thing.

“That is a perfect example of grounded theory, because we’re not looking for anything in particular. We’re not asking about anything in particular, but over time trends will emerge from those emails we get. We’ll see, ‘Oh, this is something we need to fix,’ ‘This is something we need to build,’ or ‘This is an integration that we need to connect.’

(He’s using emails as the means of data collection but the same principle is being applied.)

If you’re having trouble selling stakeholders on the user interviews, feel free to point them to the JTBD framework for uncovering and prioritizing customer needs. It’s an approach that scratches the same itch but isn’t taught as a UX-specific technique and thus may be free of any connotations the word “UX” may have. Or just do what Gregg in did and use open-ended emails to customers.

Good luck,