How do you measure user/customer experience for an in development product?

Question

This may at first sight seem an overly broad and basic question. But drill down a little.... its really not so simple.

Measuring usability- easy. Lots of different ways to test and measure the usability of a system

Getting qualitative input about user experience- also pretty easy.

Measuring overall user experience however, for something that is not even being actively used yet- is this at all possible?

Certainly for an already in existence product there are methods. Customer satisfaction surveys, interviews where you ask about parts of the process that annoy them, etc....

But for something which has not yet been launched... is there any way to get a solid, scientifically reliable measure of its user experience?

Does it require having a past version to use as a benchmark? Assuming one has this, what method is possible here?

Answer 1

1. Decide what success means

Decide on your metrics of success early. This is the product's soul, the guiding principals for everything you do. Kerry Rodden's HEART framework is a great structure for that.

2. Test incrementally

When you're developing something new (most of my time), you have to test pieces. It's not an "overall" view, but it will guide you to a wholistic experience if you're always driving toward your success metrics.

Guerrilla testing is a great tool for this. You can quickly get an idea of how close you are to the mark. When you have a collection of features that make up some sub-workflow, then you can test it a bit more formally.

You're always building toward the final, wholistic test.

3. Release something as early as possible

I'm not talking about a big splash roll out. Find a subset of your audience that doesn't need everything. Build their MVP and get the alpha program rolling. That's where the whole picture starts to come together.

There's no doubt that you missed somethings in your incremental tests. Probably a lot of things. With this small group who's offered to help you out, you can piece together the rest of the puzzle. Then you move to beta and a full release when that's done.

Throughout, you're checking components, workflows, then the whole experience against the guiding principals. Sticking to a singular mission* is the only way to predictably reach a successful overall experience.

* There's always some course correction along the way. Just don't go nuts.

Answer 2

You could try PRETOTYPING

pretotyping is a way to test a product idea quickly and inexpensively by creating extremely simplified versions of that product to help validate the premise that “If we build it, they will use it.”

Personally, I find it more a great idea than a valid scientific method (well, it doesn't pretend to be scientific), but we've played a bit with these concepts just for fun and some very valuable (and measurable) insights came out from these sessions, so maybe it's the right choice for you

Additional Reading:

• What is Pretotyping and Why is it So Great for SaaS?
• How is pretotyping different from prototyping?
• Pretotyping Cheatsheet