Obtaining metrics to prove your designs will improve the UX

Question

Say you were asked to improve the UX of an already existing feature, and you wanted to provide some kind of proof and metrics how is this approach.

1. A 'talk out loud' user test (6 participants) giving them a key task to perform and taking notes of key issues and problems.

Using things like task completion rate and time on task isn't really going to provide you with any statistical confidence with a sample size of 6?

2. Then perform something more quantitive on a much larger group (e.g. 60 participants) so rather than 'talk out loud' (which can result in a lot of video footage to go through) just give them a task to do and record task success rate and time on task, as well as something attitudinal like 'how easy or hard, did you find that task [1 - 5]'.

This is then your starting data.

Then design and prototype and iterate through designs.

For the final design create an advanced prototype and then repeat the same thing so 6 'talk out loud tests' followed by 60 tests where you're recording task completion rate, time on task and easy/hard rating.

At that point, assuming the metrics are good, you should have proven that what you designed is an improvement in UX before handing it over to developers.

Thoughts on that approach anyone? Any alternative approaches?

Answer 1

I think you're basically on a good track (word of caution for when you are presenting data: our UX studies don't prove or predict improvement, they can only provide data for direction and suggest improvement.)

Start with a goal, or at least a hypothesis of what you're trying to find out - What does success look like? What are you comparing your data to? A previous study, your competitor, a client commitment, an arbitrary guess by your stakeholders? You could use interesting findings in your analytics to kick things off. Industry benchmarks exist if you have no good data around.

I would suggest flipping the order on your tests - start with your quant unmoderated usability study (40 participants is the recommended minimum). See what the quant data tells you. Did you hit your goal? Are you way off? If you made goal, you might not need additional testing. If not, use qualitative think-aloud studies to find out what's getting in the way of success.

Then, you keep iterating and testing until you make your goal, or your hypothesis is sufficiently informed to make a decision.

Answer 2

You could do a quant prototype AB test with both your propose design as clickable prototype AND of your current design. This way you can compare results like time on task and success rate side by side to justify the effort to build this improvement.