User Experience Stack Exchange is a question and answer site for user experience researchers and experts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

After my company has recently been acquired, we have been allotted an hour of "usability testing" at an upcoming conference. The catch is, if we follow the template that has been put forth to us, it's not actually a usability study anymore, it's a focus group of 20 people in a room for an hour.

We will not be speaking directly with end users, but rather their managers etc. So while we will gain some great insight into how they envision the administration and setup of the software, so far it seems like a "tell us what you think/want" session, versus having the users show us how they use it, which is a cardinal sin in usability testing.

So far I'm thinking of opening with a 5-second test to gain initial impressions before we introduce any bias. From there, a quick demo (which is required in the session), and then perhaps having the group perform a task through consensus. I feel that this would help us see how they justify the actions and verbalize their thoughts as a group, but I am a little worried about an "alpha" participant skewing the results. And then closing with the participants explaining how this would fit into their existing system, and what features they are looking for.

Does anybody have any experience making a focus group more usability/user experience based? Any feedback or critique on my (rough) plan for guiding the session? I feel like I can't find any resources that combine the two.

I should add that while group usability testing would be great, we only have one screen (which is why I was contemplating navigation by consensus.

share|improve this question
4  
What a challenge! Could you try collect some "usability goals" from these folks instead? Rate of success, time per task, efficiency, effectiveness, etc, etc. Then use this as a basis for further "quality assurance". Letting them perform a task through consensus sounds like a good idea. Perhaps you could use the goals you have set to point out the areas that needs more attention and actual user-testing afterwards ;-) – Jørn E. Angeltveit Feb 12 '14 at 17:24
    
@JørnE.Angeltveit I think your comment addresses the question better than the accepted answer, so you should put it down as an answer for the bounty. – Michael Lai May 10 at 22:26
    
how did it go in the end? any lessons learned for other people in similar situation? – Aprillion May 15 at 9:41
up vote 2 down vote accepted

If you can actually get the users to quickly perform specific tasks rather than doing a demo that would be be helpful as you can observe their reactions and user flow and define what are the usability concerns.

If that's not feasible, One approach you can do is to define what are the different user flows you want to evaluate (it would be good if you can ensure the number of user flows you have is equal to the number of the number of participants).

Once you have the user flows defined:

  • you could ask each participant in turn to explain how they would use the system to perform it and what are the challenges they face.
  • Once you get inputs from the primary speaker,you can open the discussion for other speakers to provide inputs on what they feel can be improved or the challenges they face.
  • I know this goes back to the situation of them telling you what they want but it might also give inputs on how many people feel the same way on a certain user flow and what functionality is critical and what is the expected behavior.
  • You could also ask for anecdotes about how their users had specific complaints about the functionality and the workarounds they had to employ.

That said there are some concerns from the above method

  • Users might not be willing to talk freely or might look for other users for approval which might skew your results.
  • Users might not be able to remember the steps they take to handle the different user flows.

I would also recommend looking at this study on how Kepler used group usability testing to understand issues ( rather different from your needs but it might give some inputs)

The test included the following usability activities (in temporal order):

  1. User Profile Survey – 20-30 minutes

-Technology Expertise

-Software Inventory

-Job, Education, and General Demographics

  1. Basic Tasks Exercise – 1 hour

-Run an existing workflow, then add one output/display component. -Create a new workflow (simple graph plot of data).

  1. Usability Issues Discussion – 1 hour

These activities were embedded the training program. The user profiling activity was done at the beginning of the training workshop. As part of the training, participants were given an introduction to key terms and concepts, and also performed several familiarization exercises with the Kepler software. After this introduction to Kepler, the group usability testing was conducted.

Participants were given two basic task exercises to perform on their own. These tasks were given as a set of written instructions. Participants were told that observers would be watching and recording issues, and that they might ask quick questions. Users performed the given tasks individually, but simultaneously. The participants performed the following tasks: 1) modify an existing workflow, and 2) create a simple workflow.

Users were seated in one large training room in groups of five, with each group in a somewhat circular pod configuration. They couldn’t easily see their neighbor’s screen without purposefully leaning over to look. During testing, multiple observers walked around and interacted with participants, answering questions, and minimally probing users. The tester/observers included a usability professional, a trainer, and a software developer

share|improve this answer
2  
This answer seems to assume that the OP is going to have an hour with the users - but the question stated that there will be no users there, only their managers. They don't use the system, and don't know what challenges there are, they only know "I want my underlings to be 15% more productive because they now have data on their fingertips". – Rumi P. Feb 13 '14 at 11:32
    
20 people => 3 minutes per flow, including the discussion, sounds like too little time to me.. also not sure how productive and honest the discussions between 20 people who might see each other for the first time can get – Aprillion May 14 at 20:51

What if you made it more of a discussion about business needs and goals?

Example: In this screen users can search for X and compare it to Y. We understand this is important to your business because the difference between X and Y is often a place where cost savings can be identified. Do you agree this is an important part of your business?

Another example: In this screen you can see rankings between sales people by region. We provided this view because we were told regional ranking is the most important on a day-to-day level. Is that true or is there a different ranking you use more than regional?

In this way the managers can provide business objectives, which is what they know more about than tasks and screens -- and certainly more valuable than any UX "guidance" they may want to provide.

share|improve this answer

Given your situation, any testing that you do will create an extra layer of assumption (i.e. that the managers think like the end-users), which you may need to validate or else it is very likely for you to fall into the trap of designing something that you can sell to the managers but the users won't want to use. So to me it is possible to get useful data from a focus group even if it is just the managers, but you have to be able to do some validation of the data that you collect.

How can you do this without end-user involvement? I think a possible angle or approach to take is to highlight to the managers the difference in the way they think compared to the users, but for this to happen you will need to have some information from the users. I think one way to try and obtain this information is to get the managers to ask the end-users on your behalf. For example, let them list the three most important things they think needs to be included, and then ask the end-users the same question and get them to compare the differences.

If you can't even do something like that, I would perhaps get user data from a similar usability testing session and get the managers to go through the same exercise and compare the difference.

Failing that, I don't think it makes much sense to do testing that you can't validate on assumptions that you don't think will be correct - not a particularly sensible way of doing UX design.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.