User testing: asking experienced users to bring frustrating test cases?

Question

Is it a valid testing strategy to ask experienced users to bring their own frustrating use cases to the table, rather than giving them our own task to work through?

Are there caveats or best practices for that kind of test?

The case I'm working with is a new Find feature that we want to evaluate and perhaps tweak internally before releasing more broadly. We also want to think about how to explain the tool and what it's good for in our "what's new." Our application is a complex forms-based document (kind of like Turbo Tax) that is used to write proposals for astronomical observing on space-based telescopes.

Here's text from a draft email:

In advance of our meeting, we would like you to think of two or three tasks that were challenging or frustrating in APT: either understanding a proposal that came across your desk, or a difficult or time-consuming editing task. We'll be having you install a new beta version that contains a new Find tool and seeing whether that tool helps you address the challenges you brought. Please try to have the relevant proposals available to load into the new beta.

Answer 1

I would say that this is absolutely a good idea.

Presumably the reasoning behind it is that you aren't sure about the real-world use cases and workflow that might cause problems. If that is true, you need to do everything you can to find out, and asking the users to bring testing scenarios sounds like a good approach.

The real risk would be to conduct testing that "passes" but omitted issues that were important to users because your test cases were incomplete.

Maybe an argument can be made that you should have captured this information earlier, and thus it shouldn't be part of the testing. However,

• Given that you have gotten to this point and still need more information on how users will use the product, this is a good way to get it.
• The requirements gathering process is never perfect, so even if things are "done right" from the beginning, there is still room to learn more.

Answer 2

I think this is a very good idea. I think you should even consider expanding your audience beyond experienced users by including novice and intermediate users.

No matter how hard I try to come up with every action or set of actions a user might choose, with every new app I work on, the users astound me with the crazy things they will try.

If you really want to see how stable and intuitive your new design is, novice users will really beat it up. If you can accomplish everything that the experienced users need and can make it unbreakable by the novice users, you have a winner.