A/B Testing vs Cohort Analysis

Question

I had a few questions about A/B Testing vs Cohort Analysis (dividing users based on registration time).

1.) Is A/B testing mainly used for UX design changes (such as change in Sign Up button Color) or can be used to split test features (such as conversion rate on freemium vs free trial)

2.) What conclusions can I draw from results received from A/B Testing vs Cohort Analysis? Example: A/B Testing helps to tell us that this feature improves conversion in general irrespective of the user Vs Cohort Analysis tell us that conversion of users joining in month 2 is better than in month 1 gives us overall direction a site is going in (conversion rate may or may not be because of new feature implemented between month 1 and 2).

3.) Is it true that A/B testing can be used to test hypothesis faster and Cohort Analysis takes more time?

4.) Should we run A/B testing on a new feature to see if conversion rate has improved and then use Cohort Analysis to see long term effect of new feature? (if Cohort Analysis shows improvement in conversion month over month how strongly can we attribute it to the new feature implemented if no other changes are made to site)

Answer 1

The point of doing A/B testing rather than cohort is that it eliminates the conflating variable of time. The data you gather is only valid if there aren't ulterior explanations for why two groups behave differently, and groups doing things at different times will often behave differently.

For example: if you're an e-commerce site and you compare user behaviors between people who registered in December vs. people who registered in January, you're going to get wildly different results due to the Christmas shopping season. This would totally invalidate a cohort analysis, unless your goal is to observe how these two groups behave differently.

In short, A/B testing is for testing how two features are different. Cohort analysis is for testing how two groups of users are different.

1. You can A/B test anything you want! The only requirements are that the two sides (a) happen simultaneously and (b) are randomly assigned.
2. Again, A/B tells you how features differ in effectiveness and cohort analysis tells you how groups of users differ in behavior.
3. A/B is faster if your cohort analysis requires you to wait for new people to sign up. But if you're just breaking existing users into blocks, it should take the same amount of time.
4. This is a reasonable approach, but remember that at the end of the day, the research method you apply needs to be based on the kind of information you're looking to gather.