0

I intend to do quantitative user tests of an existing tool and compare its usability performance to a new design that I'd propose.

Let's say after doing user research I come up with task scenarios (based on real needs) that include goals which can't be achieved with the current tool.

How does one proceed in that case?

It'd surely feel wrong to test a scenario with an unachievable goal. It would also feel wrong to test different scenarios for the current and new design (while also eliminating comparability).

Would a valid approach be defaulting to a 0% success rate for such task scenarios? Or would it be better to leave out such scenarios completely and only test tasks that have achievable goals in both, the old and new design?

  • Wouldn't it make more sense to make test scenarios which can be applied to both designs, if you want to compare them and then perform a separate user test to evaluate the new feature which the old design lacked? It feels like you are trying to do two things at once. – DuKes0mE Apr 29 '18 at 21:03
1

Let's say after doing user research I come up with task scenarios (based on real needs) that include goals which can't be achieved with the current tool. How does one proceed in that case?

Assuming this need you mentioned is in the product roadmap, I would use those new task scenarios to try and test different concepts against each other. Don't use those tasks for new vs. legacy usability testing.

Or would it be better to leave out such scenarios completely and only test tasks that have achievable goals in both, the old and new design?

If you're comparing old and new on the basis of usability, correct - leave out such scenarios that aren't supported on both. Only use tasks that have a success case in both old and new. Otherwise it's not a valid comparison.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.