How can I get useful usability data from a focus group that are not made up of end-users?

Question

After my company has recently been acquired, we have been allotted an hour of "usability testing" at an upcoming conference. The catch is, if we follow the template that has been put forth to us, it's not actually a usability study anymore, it's a focus group of 20 people in a room for an hour.

We will not be speaking directly with end users, but rather their managers etc. So while we will gain some great insight into how they envision the administration and setup of the software, so far it seems like a "tell us what you think/want" session, versus having the users show us how they use it, which is a cardinal sin in usability testing.

So far I'm thinking of opening with a 5-second test to gain initial impressions before we introduce any bias. From there, a quick demo (which is required in the session), and then perhaps having the group perform a task through consensus. I feel that this would help us see how they justify the actions and verbalize their thoughts as a group, but I am a little worried about an "alpha" participant skewing the results. And then closing with the participants explaining how this would fit into their existing system, and what features they are looking for.

Does anybody have any experience making a focus group more usability/user experience based? Any feedback or critique on my (rough) plan for guiding the session? I feel like I can't find any resources that combine the two.

I should add that while group usability testing would be great, we only have one screen (which is why I was contemplating navigation by consensus.

Answer 1

If you can actually get the users to quickly perform specific tasks rather than doing a demo that would be be helpful as you can observe their reactions and user flow and define what are the usability concerns.

If that's not feasible, One approach you can do is to define what are the different user flows you want to evaluate (it would be good if you can ensure the number of user flows you have is equal to the number of the number of participants).

Once you have the user flows defined:

• you could ask each participant in turn to explain how they would use the system to perform it and what are the challenges they face.
• Once you get inputs from the primary speaker,you can open the discussion for other speakers to provide inputs on what they feel can be improved or the challenges they face.
• I know this goes back to the situation of them telling you what they want but it might also give inputs on how many people feel the same way on a certain user flow and what functionality is critical and what is the expected behavior.
• You could also ask for anecdotes about how their users had specific complaints about the functionality and the workarounds they had to employ.

That said there are some concerns from the above method

• Users might not be willing to talk freely or might look for other users for approval which might skew your results.
• Users might not be able to remember the steps they take to handle the different user flows.

I would also recommend looking at this study on how Kepler used group usability testing to understand issues ( rather different from your needs but it might give some inputs)

The test included the following usability activities (in temporal order):

1. User Profile Survey – 20-30 minutes

-Technology Expertise

-Software Inventory

-Job, Education, and General Demographics

1. Basic Tasks Exercise – 1 hour

-Run an existing workflow, then add one output/display component. -Create a new workflow (simple graph plot of data).

1. Usability Issues Discussion – 1 hour

These activities were embedded the training program. The user profiling activity was done at the beginning of the training workshop. As part of the training, participants were given an introduction to key terms and concepts, and also performed several familiarization exercises with the Kepler software. After this introduction to Kepler, the group usability testing was conducted.

Participants were given two basic task exercises to perform on their own. These tasks were given as a set of written instructions. Participants were told that observers would be watching and recording issues, and that they might ask quick questions. Users performed the given tasks individually, but simultaneously. The participants performed the following tasks: 1) modify an existing workflow, and 2) create a simple workflow.

Users were seated in one large training room in groups of five, with each group in a somewhat circular pod configuration. They couldn’t easily see their neighbor’s screen without purposefully leaning over to look. During testing, multiple observers walked around and interacted with participants, answering questions, and minimally probing users. The tester/observers included a usability professional, a trainer, and a software developer

Answer 2

What if you made it more of a discussion about business needs and goals?

Example: In this screen users can search for X and compare it to Y. We understand this is important to your business because the difference between X and Y is often a place where cost savings can be identified. Do you agree this is an important part of your business?

Another example: In this screen you can see rankings between sales people by region. We provided this view because we were told regional ranking is the most important on a day-to-day level. Is that true or is there a different ranking you use more than regional?

In this way the managers can provide business objectives, which is what they know more about than tasks and screens -- and certainly more valuable than any UX "guidance" they may want to provide.

Answer 3

Given your situation, any testing that you do will create an extra layer of assumption (i.e. that the managers think like the end-users), which you may need to validate or else it is very likely for you to fall into the trap of designing something that you can sell to the managers but the users won't want to use. So to me it is possible to get useful data from a focus group even if it is just the managers, but you have to be able to do some validation of the data that you collect.

How can you do this without end-user involvement? I think a possible angle or approach to take is to highlight to the managers the difference in the way they think compared to the users, but for this to happen you will need to have some information from the users. I think one way to try and obtain this information is to get the managers to ask the end-users on your behalf. For example, let them list the three most important things they think needs to be included, and then ask the end-users the same question and get them to compare the differences.

If you can't even do something like that, I would perhaps get user data from a similar usability testing session and get the managers to go through the same exercise and compare the difference.

Failing that, I don't think it makes much sense to do testing that you can't validate on assumptions that you don't think will be correct - not a particularly sensible way of doing UX design.