If you can actually get the users to quickly perform specific tasks rather than doing a demo that would be be helpful as you can observe their reactions and user flow and define what are the usability concerns.
If that's not feasible, One approach you can do is to define what are the different user flows you want to evaluate (it would be good if you can ensure the number of user flows you have is equal to the number of the number of participants).
Once you have the user flows defined:
- you could ask each participant in turn to explain how they would use the system to perform it and what are the challenges they face.
- Once you get inputs from the primary speaker,you can open the discussion for other speakers to provide inputs on what they feel can be improved or the challenges they face.
- I know this goes back to the situation of them telling you what they want but it might also give inputs on how many people feel the same way on a certain user flow and what functionality is critical and what is the expected behavior.
- You could also ask for anecdotes about how their users had specific complaints about the functionality and the workarounds they had to employ.
That said there are some concerns from the above method
- Users might not be willing to talk freely or might look for other users for approval which might skew your results.
- Users might not be able to remember the steps they take to handle the different user flows.
I would also recommend looking at this study on how Kepler used group usability testing to understand issues ( rather different from your needs but it might give some inputs)
The test included the following usability activities (in temporal
- User Profile Survey – 20-30 minutes
-Job, Education, and General Demographics
- Basic Tasks Exercise – 1 hour
-Run an existing workflow, then add one output/display component.
-Create a new workflow (simple graph plot of data).
- Usability Issues Discussion – 1 hour
These activities were embedded the training program. The user
profiling activity was done at the beginning of the training
workshop. As part of the training, participants were given an
introduction to key terms and concepts, and also performed several
familiarization exercises with the Kepler software. After this
introduction to Kepler, the group usability testing was conducted.
Participants were given two basic task exercises to perform on their
own. These tasks were given as a set of written instructions.
Participants were told that observers would be watching and recording
issues, and that they might ask quick questions. Users performed the
given tasks individually, but simultaneously. The participants
performed the following tasks: 1) modify an existing workflow, and 2)
create a simple workflow.
Users were seated in one large training room in groups of five, with
each group in a somewhat circular pod configuration. They couldn’t
easily see their neighbor’s screen without purposefully leaning over
to look. During testing, multiple observers walked around and
interacted with participants, answering questions, and minimally
probing users. The tester/observers included a usability
professional, a trainer, and a software developer