How many months should we track and collect data before analyzing it after rolling out a new feature?

Question

I'm a solo product designer, and this is my first time to create and apply UX metrics and KPIs in our product as well as doing UX research. So far, our company doesn't have a budget yet to hire an expert in UX research but there will be a possibility if they find value in UX research.

When I created the design, I haven't performed usability test before the developers work on it (mainly because we're still in the process of applying or convincing the company to invest in UX research but we're planning to implement that soon too.) Because of this, I plan to implement UX metrics or KPIs and I want to track and collect data for our new feature for our existing site. I'm also planning to use Google Analytics and Hotjar to track, collect and analyze data.

So, I want to know how long or how many months we should usually track and collect data for a small new feature? Mostly, I want to learn if there are engagement in the new feature, the task success rate, completed time task, if there are abandonment on the forms, etc. I want to show those data to the stakeholders so we know what are the areas to improve for the next iteration of the feature.

Let me know if you need further details to help me with this and I'm happy to provide. Hope someone can help me with this. Thanks in advance!

Answer 1

Important question!

I'm going to start with the typical UX designer's answer: It depends.

Some thoughts to help you come to a decision:

• initial reactions to changes (be it "uuh, shiny" or "urgh, why did they broke my workflow") can stay for a long time, but they will eventually fade. I don't know if there is data on this, but intuitively I would say you may have to ride out these effects for at least 12 months.
• Tracking continuously may be worth much more than any set amount of time. Why not have a system that notifies you when any KPI crashes or surges, independently of rolled out features? May happen, will never not be interesting. So, forever?
• At last: Don't put too much confidence in hard quantitative data when UX is the concern. You need a really good understanding of statistics (and so do your superiors) to actually analyze the data correctly. You need to keep a million biases in check. You need to be a really, really good UX designer to intuit what to measure and how. You introduce another system that can fail. And so on. Thus, handle with care. In short, listen to users, don't measure them.

Hope that helps :)

Answer 2

Track and collect data continuously. Note any seasonal shifts.

For when to measure and report out again, Nielsen/Norman group bases its recommendation on how frequently your product is accessed:

After your redesign is launched, measure your design again. There is no hard and fast rule on how long to wait after a design is launched to measure again. If you’re tracking analytics, there’s added benefit of continuous measurement. However, for task-based data collection, like quantitative usability testing and surveys, you’ll need to determine the right time to collect the data. Users often hate change, so give them a bit of time to adapt to the redesign before measuring it. The amount of time varies depending on how frequently users accesses your product. For products accessed daily, perhaps 2–3 weeks is enough time. For a product that users access once or twice a week, 4–5 weeks before you measure is better.

Answer 3

I think there are some things to consider.

Collecting data is trivial (you already said you'll use GA). You won't need to save anything; data is in the cloud, so you can generate reports as needed.

Now, about a new feature: you can use regression analysis or ANOVA to measure this, and you won't even need to test with users. Unless your feature is detrimental to the existing features, the worst-case scenario is a status that equals 0 (this article on Quantum UX may be of help).

Quantum UX Energy is a fundamental concept of the QUX paradigm: it is what enables UX to acquire this Q, so to speak. In other words, it is multiple information flows that set a static system in motion.

To understand it better: Let’s take a web page. Let’s say we see it in its initial state, the day we launch it, before anyone else sees it. This page will have an energy level (QUXE) of 0.

But once we observe its behavior and performance, we get data that allows us to make decisions.

For example:

• increase or decrease the amount of advertising purchased
• reposition elements to do CTR tests
• develop engagement strategies
• create marketing strategies with multiple products
• define geographically localized strategies
• analyze results and create probabilistic predictions

As we can see, all the usual things that any more or less serious company does on a daily basis, with tools that are available to everyone. And just in case we’re talking about free analytics tools .

So once we have the data from multiple sources, at different times, with different variables, we can take action.

In other words, the initial static element will have different types of interactions; from stakeholders, from users, from dynamic content scripts, from advertisers, and so on.

Therefore, it is logical to say that the static element already has an energy charge that it did not have at another point in time. This energy can be represented with formulas and equations. And at the same time becomes part of other equations that allow us to generate more efficient probabilistic systems than those we could develop by looking at the data.

Technically, you can add a new feature every day, but it will be quite complex to measure its effects on the app, and of course, it will confuse users. However, adding a feature every now and then is a good idea because you're signaling to users that your app is under continuous development. More likely than not, engagement will be greater than zero.

In short: as long as you measure the effects, you'll be fine. Nonetheless, if you can't test with users before launch, I'd highly advise conducting very thorough QA.

Answer 4

The answer typically is "yes". Collecting data is exceptionally cheap, and a good indicator of stuff breaking. For example, if the line has been at a steady 80% success rate and continues to be for the next 8 months, if it suddenly dips to 30% or 0% you know that something broke horribly.

What typically isn't necessary is to retain this data forever though: Chances are that in a year nobody will care how the feature performed at launch, so retaining data for much longer than a year generally isn't necessary.

Answer 5

In addition to the other answers: it depends a lot on your traffic volume.

If you have millions of visits every day which all use this feature, you'll notice changes quite quickly. If you only have a handful, sure, it can take months or years before you have enough data to know anything for sure.

Note that there can be other factors than just your changes which affect results (day-of-week, holidays, other seasonal variations, sales, other changes...), so ideally you should perform A/B testing, where the old and new versions are shown to different people, and you can then compare the results for each population on the same timespan. Depending on the proportion of old and new, this will increase the time you need to get significant results more or less, but it gives you a better comparison.

Of course, if the KPI you want to measure is conversion, or return rate, then you have to take into account the delay for that. If people take a month from first visit to sale and the change you made is something they'll see on their first visit rather than at the very end of the sales process, you can't have results in a week.