How can I capture quality data about existing users' goals, outside the context of UI?

Question

We are rebuilding a large web application from the ground up. We have several thousands of technical users currently managing digital products in a web portal with a complex account/user structure. It has security overhead and a legacy nature that complicates the process of installing third party analytics tools. The UI is a mess of bolted on, unscalable functionality, and so we're completely reconsidering almost every aspect of the platform.

I'm putting together a strategy for reaching out to existing users to validate our assumptions from the persona development process, and to attempt to work their actual goals into the process from even before the ideation phase. I need a communication process that provides optimal data capture, and that allows sufficient control of the conversation to keep it focused on their goals, outside the context of any UI.

The key points from a user interaction perspective are that I don't have anything visual to share about the new UI - this isn't user testing - and I need to ensure the conversation isn't hung on the hooks of the problems with the current system.

Does anyone have experience in managing this type of communication and implementing it in an agile development process? What type of communication tools did you use, and what are the lessons you've learned?

Answer 1

You might find this article about interviewing users to be helpful.

Decide how much you can share with the users. Can you mention that you are reworking the interface? You might find it helpful to be blunt, that you are aware of usability/design issues with the current interface, and are reworking the system. To make sure you focus on the right tasks, would they mind answering some questions?

You may also have to listen to their existing frustrations with the software. But asking the right follow up questions should help validate your current/future efforts. Get them to tell you about their experience.

For example, if they complain about the existing interface:

• ask what they were trying to do when they encountered the issue. "Tell me a time when you ran into that issue. What was your goal and how did that interfere with it?"
• ask more whys, why was that task important? What was the intended outcome of the task?
• how often did they encounter the issue, was it a task they performed daily?

You may not be able to avoid hearing about current problems. But listen, and dig deeper, and you should find out a lot without getting stuck.

Additionally, don't forget to ask what the system does well! You don't want to leave out existing appreciated functionality!

Answer 2

Since you mention that the system is very complex and difficult to add analytic tools, you will have to work with what you have, what do you have?

One thing you have is web server logs, start there:

• Pages that are loaded the most, that indicates what is more useful in terms of content or indexes.
• Time spent per page, that indicates problematic pages, content that is difficult to understand or quality content.
• Response time, this gives you slow pages, which can be because database or programming problems.
• Redirection or processing rules that consume time and resources, with that you gain the knowledge to improve folder structure, navigation and common typos.
• Pages that are less visited, that gives you an idea of useless pages or hidden resources that should be brought to attention.
• Referrer. That gives you an idea if users are typing addresses (or copy and paste) which indicates problems to find the pages or bookmarked pages.

Most of the options above have a good side and a bad side to interpret, deciding which one is the right one each time depends on your knowledge of the system itself, the users, their goals and the logs.

You should combine that with some kind of interviewing or online questioning to help you decide. If you can't, then just use the above information.

Another thing that you have is, most probably, server space, so once you have the basics of the new system, build an interface and make that a live test, ask users to go to that beta site and use it, add a simple feedback form on each page where they can, anonymously, tell you what they want and what they dislike. Combine again that input with the information from the logs of the old site, the logs of the new site and any analytics that you set on the beta site. If you do this, before asking users to test it, you have to be sure that there are no serious problems, if users find holes, closed doors, unexpected behaviours, cryptic error messages, etc, they won't like the new site and most of the input will be bad an inaccurate.

The most important aspect, I'd say, is to build the new system in a flexible way, so you can adapt it quickly after a short period of testing, either on a beta site or in a final, live site. Consider that users will give you input at some point and you have to use it, if the site is too rigid, then changes are going to be too difficult to implement and then user input will be, most probably, ignored.

Answer 3

In a nut shell it's called user research. From user research you don't get data but information.

The most direct form of user testing are one to one interviews. Find out what they do, why the do it and what they need to do it. Never ask them what they want directly, don't lead the questions and play dumb if they ask you questions about what your company is doing in the future!

Do a search for contextual enquiry as that's the technical name for one to one user research sessions.

Also try to avoid 'assumptions' in any personas you use. The more made up stuff in a persona the more likely they are to lead to the wrong solution.