Did not perform A/B Testing , how to define success of feature

Question

The problem is regarding a educational matching platform. We match Learners together so they and the fellow learners can video call. Learners are entered into a ‘matching pool’ where they can see all their potential matches, and then they go through and send requests to other learners. An example of a feature/attribute that we want to test is: We recently allowed the capability for learners to write a short bio about themselves that we will include in their info section when other learners are in the match pool sending requests (previously it was simple info like country name, availability schedule, etc.). We want to see whether this addition of adding the bio will increase the number of match requests that teachers send.

Whenever one needs to analyze the success of a certain feature, we need to always compare a cohort who used the new feature versus a cohort who did not use the new feature. The best way to do this is through A/B testing, but because of our small team this was not done. Hence all users were given access to the feature.

But now faced with a different problem for measuring the success of this feature-- what do we compare them to? What timeframes do we use?

Answer 1

We'd need more context here. What problems existed before the feature? What problem is the feature trying to solve? Based on the metrics around that problem, you can move forward with testing the feature. You can test whether the feature solved the problem and also create an different feature solving the same problem and A/B test both.

I'm assuming because of your small team, you are forced to cut corners for rapid production. However, it's never too late to get user feedback and testing.

Answer 2

As @jeancode said, it greatly depends on the aim you had when this feature request has been made.

As an example partially based on you use case, say that your aim is increase the engage time, the amount of time two learners that match spend messaging and exchanging informations.

The first step I would take is to quantify the objective, so lets say that we measure the engage with the number of messages exchanged per day (I'm assuming the main interaction is through an internal messaging system.)

Then you could measure the performances of the system by averaging the number of messages per couple of learners over all the learners of the system. This would give you a quantitative measure you could use to compare the performances before and after the roll out of the feature.

Beware that this is a toy example, there are a lot of things to consider before finding and effective measure of what you want to improve and use that.

In your case, since you cannot A/B test, the only constraint you have is finding a measure that you can calculate a posteriori, with data you already have.

Answer 3

Do you have the ability to reach out and try asking your users directly? Sending a quick "Thanks for using our service" email, asking if you could arrange a 5 min Q&A or just asking a few questions online will help. What information helped you select your match? Is there any information you think would be beneficial to selecting a match in the future?

Numbers are great and you should always do your best to capture whatever metrics you can but don't underestimate the power of one positive user quote when justifying or trying to get buy-in for a feature.