How do you A/B test a ranking algorithm?

Question

Let me give a hypothetical situation on A/B testing a ranking algorithm. Say I'm in charge of the popularity ranking algorithm on Youtube. I need to surface the most interesting videos to top of the list. I have two algorithms. Perhaps they are:

• PopularityA = 10 * views + 5 * comments - 2 * secondsOld
• PopularityB = 8 * favorites + 2 * likes + 4 * views

Half the users on Youtube will use PopularityA and the other half will use PopularityB. How will I quantitatively measure the happiness value from each test group to find out which algorithm is better?

Answer 1

I think the problem is you need to define your success criteria. You say that you want to get the most "interesting" results at the top of the list. So how do you define this? Maybe you should look at how many people click how many of the top 10 results, if "clicking on a video" is a good enough definition of "interesting". If your definition is that they enjoy the video enough to comment on it, then count the number of comments off videos linked to from the list.

I think that once you have clearly identified what your ranking algorithm is actually ranking for, then you will have your comparison method. As it is, the problem is that your definition is still too wooly.

Answer 2

You need to define what happiness is. Is it the fact that the person found the video he had been looking for or he was happy with the results returned by the search query? What would be the case, your determination for "happiness" would be at the search result level and not on the video page since the "happiness" factor there would be influenced by how much he liked/disliked the video.

So an approach you can take is similar to what amazon does with their review system:

where you ask users if they found what they were looking for. Of course, you can go a step further and ask them to provide anonymous comments about how they liked the search results.

Another approach would be to present a rating system near in the search results page which allows users to rate the quality of the search.

While these might provide you with some inputs about the quality of the search,they are very dependent on the user noticing the feedback/rating option. So another approach you can take is to see how many people actually select one of the videos on the search page and go to the video page instead of entering another search query. Another way of drilling down into this is to see the success of your search results would be to add a weight to each video (e.g. the top video would be the most weighted, the second video the second highest weighted and so on)

If you really want to drill down even more,you could do a measure of how much time the user spent watching the video before returning to the search results to try another option to see the level of satisfaction.

---

Now, I know a lot of this is theoretical and there are a lot of buts and if's but hey you asked for the solution of a theoretical question :)

Answer 3

It all depends on your goals and the type of the list you're creating.

Some lists, such as popular and latest, are constant to all users at the particular time period. Users don't care about your algorithm there because such lists are perceived as facts: a list of most recent content or the list of content with the most views/comments/likes/combination/etc in a time period. In such lists, it's completely up to you to decide which factors are more important for the rank: give more/less weight to the reactions you want to showcase (i.e. views, comments, likes, shares, etc.).

When you're dealing with lists that depend on user information, such as search results to a query or recommendations based on historical usage/profile/connections/etc, ranking is far more important. Yet, there are many ways to measure success. It can be the number of people who:

• see only 1 of the first 5 items presented & don't check out the rest;
• try all 5 top items presented and don't modify the query;
• perform some action on the content page (save, rate, upvote, comment, etc);
• don't visit the content page but perform an action from the page with the list.

The decision of what measure to use is completely up to you because it depends on your conversion funnel & business goals.

However, I would warn against relying only on self-reported surveying (e.g. asking "Was this helpful to you?") because they will depend greatly on the popularity of content and/or the feature. Take a look at this screenshot for the top review of Kindle Fire on Amazon:

In 4 months of its existence (as of March 8, 2012), 22,108 people of the millions who have visited the page have voted on its quality. If that's not striking, here's the top review for Fight Club (the book)

In the 9.5 years (as of March 8, 2012), only 398 people of hundreds of thousands have voted on its quality. That's very low.