How broad or narrow to go with a test hypothesis for comparing old designs with a redesigns

Question

Say I am given an open brief like 'improved the sign-up journey for product X' or 'improve the search experience for product Y' etc.

I want to do some user testing around the current experience so this will be the baseline metrics and when I user test my designed solution I can compare the old with the new.

Is a general test hypothesis like the following too broad?

We believe that by redesigning the sign-up journey we can improve the overall sign up experience We will know this to be true when we increase the success rate, decrease time on task and increase the attitudinal score for happiness (1 - 5) and ease (1-5)?

Or is it better to break down hypotheses like:

We believe that by ? simplifying/reducing the number of steps? in the sign-up journey, we can reduce time on task We will know this to be true when the mean time on task is lower on the redesign

I see a floor in the latter. Your opening hypothesis, on which will form your benchmark tests, is assuming that reducing the number of steps in the sign up journey is the right solution but you've barely begun your discovery phase so maybe this is wrong?

Answer 1

How did you get to this hypothesis in first place? Always better to be as specific as possible. This is a framework to get to a good hypothesis:

1. Gather insights

The first step for a redesign / improve a feature is to understand the problem you are trying to solve. You need to gather data and insights. This might come from qualitative or quantitative methods.

2. Create an opportunity tree

The second step is to create opportunities and problem statements based on those insights.

"Based on report x we found out that x% of users drop from the sign up process because they get overloaded with some many form fields"

3. Write your hypothesis

Hypothesis A

x% of users drop from the sign up process because they get overwhelmed with some many form fields. By splitting the sign up flow into 3 steps, we believe that will reduce cognitive load and will reduce drop outs by x%.

Hypothesis B

x% of users drop from the sign up process because they get overwhelmed with some many form fields. By reducing the amount of information we ask users during the sign up process, we believe that will reduce cognitive load and will reduce drop outs by x%.

4. Create solutions

Then you come up with solutions that fit those hypothesis.

5. Test

You test those solutions using one or more research methods like usability tests, A/B testing, etc.

6. Learn and iterate

Have a look a Spotify's execution framework: https://spotify.design/article/from-gut-to-plan-the-thoughtful-execution-framework