How do you know what to test at the right time on a product?

Question

You have a hypothesis of a new feature that would improve x on your product and you don't have any data backing up your decision as this feature is new territory. How do you know this hypothesis is better to test now than other hypothesis, and what is the best way to test it without spending any time coding or even designing it?

Do you put a fake button to see if people are interested in this? Do you sent an email and see how many people interact to see what is upcoming? Do you make people test a website that already has this to see how they react?

I feel like doing user interviews for qualitative data in this case can be misleading because if 6 participants say that they would use the feature it doesn't mean that the rest of the people will.

So the question is, how do you know what to test at the right time so you can move fast?

Answer 1

You said you don't have any data backing up your decision. None? No data whatsoever? Maybe there is some data if you go looking? If you literally have none, how can you possibly go ahead - unless it's really a case of do it in a day, throw it out there, and see what happens.

Joe Leech's decision diamond has 4 dimensions

• Qualitative User research - research insight, customer request, unmet user need, jobs users do
• Data - analytics, trends, a/b tests
• Business value - alignment with strategy, goal, metrics, vision
• Gut feel - feels right, SME opinion, emotion, stakeholder push. How many people's gut feel?

As Joe says

If the answer to all four dimensions is a yes, build that next. Does it meet three of the four? Then build that right after. Two dimensions, further down the priority order. Only one dimension is a yes? Either park it or investigate further in one of the other dimensions. Run some qual or quant research.

If there is only one dimension with a yes, don’t do it. Especially if that dimension is gut feel.

Reversibility

Joe also some advice elsewhere about whether trying something out is a reversible thing or a one way door. If it's reversible - there's more inclination to just give something a go.

Go on 80%

Joe also says you don't have to be 100% confident to go ahead with something. If you're 70/80% confident then that's probably good enough. If you wait until you're 90% confident you're probably being too slow and losing your edge.

Also - don't rely on what people say :) If 6 participants say they'd use a feature, that doesn't mean any of them will, let alone the others!

https://mrjoe.uk/the-decision-diamond-a-simple-and-effective-prioritisation-technique/

Answer 2

Really good question! I used to have this conundrum for a while and then I found the solution in the Google Design Sprint.

You can find more information about it online but basically, the methodology answers your question exactly : "What is the the next step right now?"

It was created by Google Ventures (the company that acquires new startups for Google and helps them get on the right track) and now the methodology is open to use by anyone.

In a nutshell: In 4 days you bring in all your stakeholders and you run a series of exercises: Monday - Mapping and Sketching, Tuesday - Decisions and Storboarding, Wednesday - Prototyping and Thursday - Testing. In one week you can clarify the vision of your organization and know what the next step is.

You should try it! It works wonders and it saves so much time!