Is it a valid testing strategy to ask experienced users to bring their own frustrating use cases to the table, rather than giving them our own task to work through?
Are there caveats or best practices for that kind of test?
The case I'm working with is a new Find feature that we want to evaluate and perhaps tweak internally before releasing more broadly. We also want to think about how to explain the tool and what it's good for in our "what's new." Our application is a complex forms-based document (kind of like Turbo Tax) that is used to write proposals for astronomical observing on space-based telescopes.
Here's text from a draft email:
In advance of our meeting, we would like you to think of two or three tasks that were challenging or frustrating in APT: either understanding a proposal that came across your desk, or a difficult or time-consuming editing task. We'll be having you install a new beta version that contains a new Find tool and seeing whether that tool helps you address the challenges you brought. Please try to have the relevant proposals available to load into the new beta.