User Test with Task Scenarios That Have Unachievable Goals?

Question

I intend to do quantitative user tests of an existing tool and compare its usability performance to a new design that I'd propose.

Let's say after doing user research I come up with task scenarios (based on real needs) that include goals which can't be achieved with the current tool.

How does one proceed in that case?

It'd surely feel wrong to test a scenario with an unachievable goal. It would also feel wrong to test different scenarios for the current and new design (while also eliminating comparability).

Would a valid approach be defaulting to a 0% success rate for such task scenarios? Or would it be better to leave out such scenarios completely and only test tasks that have achievable goals in both, the old and new design?

Answer 1

Let's say after doing user research I come up with task scenarios (based on real needs) that include goals which can't be achieved with the current tool. How does one proceed in that case?

Assuming this need you mentioned is in the product roadmap, I would use those new task scenarios to try and test different concepts against each other. Don't use those tasks for new vs. legacy usability testing.

Or would it be better to leave out such scenarios completely and only test tasks that have achievable goals in both, the old and new design?

If you're comparing old and new on the basis of usability, correct - leave out such scenarios that aren't supported on both. Only use tasks that have a success case in both old and new. Otherwise it's not a valid comparison.