The problem is regarding a educational matching platform. We match Learners together so they and the fellow learners can video call. Learners are entered into a ‘matching pool’ where they can see all their potential matches, and then they go through and send requests to other learners. An example of a feature/attribute that we want to test is: We recently allowed the capability for learners to write a short bio about themselves that we will include in their info section when other learners are in the match pool sending requests (previously it was simple info like country name, availability schedule, etc.). We want to see whether this addition of adding the bio will increase the number of match requests that teachers send.
Whenever one needs to analyze the success of a certain feature, we need to always compare a cohort who used the new feature versus a cohort who did not use the new feature. The best way to do this is through A/B testing, but because of our small team this was not done. Hence all users were given access to the feature.
But now faced with a different problem for measuring the success of this feature-- what do we compare them to? What timeframes do we use?