Let me give a hypothetical situation on A/B testing a ranking algorithm. Say I'm in charge of the popularity ranking algorithm on Youtube. I need to surface the most interesting videos to top of the list. I have two algorithms. Perhaps they are:

  • PopularityA = 10 * views + 5 * comments - 2 * secondsOld
  • PopularityB = 8 * favorites + 2 * likes + 4 * views

Half the users on Youtube will use PopularityA and the other half will use PopularityB. How will I quantitatively measure the happiness value from each test group to find out which algorithm is better?

  • Please update the question to reflect that this is a ranking formula and not actually a sorting algorithm.
    – Dan D.
    Mar 8 '12 at 10:59
  • I've been thinking a lot about this question but really can't come up with a useful suggestion for AB testing this situation. I think you may have to select a different method, such as an actual usability study involving real people that you can talk to to find out if what you presented to them met the criteria. You're basically trying to test the relevancy of a generated list of results, and the only way to determine this is to speak to the people being tested to find out if the results they received made them happy, which therefore infers that the results were relevant.
    – JonW
    Mar 8 '12 at 12:52

I think the problem is you need to define your success criteria. You say that you want to get the most "interesting" results at the top of the list. So how do you define this? Maybe you should look at how many people click how many of the top 10 results, if "clicking on a video" is a good enough definition of "interesting". If your definition is that they enjoy the video enough to comment on it, then count the number of comments off videos linked to from the list.

I think that once you have clearly identified what your ranking algorithm is actually ranking for, then you will have your comparison method. As it is, the problem is that your definition is still too wooly.

  • That's my main question - to define a success criteria. I'm asking the people here how to measure the success of a ranking algorithm.
    – JoJo
    Mar 8 '12 at 16:09
  • 2
    But that is also my point, that you need to know what is important in the ranking for you. Why are you ranking them? What is your criteria for identifying a successfull process. Them we can help you to find out how to record this. Mar 8 '12 at 17:09

You need to define what happiness is. Is it the fact that the person found the video he had been looking for or he was happy with the results returned by the search query? What would be the case, your determination for "happiness" would be at the search result level and not on the video page since the "happiness" factor there would be influenced by how much he liked/disliked the video.

So an approach you can take is similar to what amazon does with their review system:

enter image description here

where you ask users if they found what they were looking for. Of course, you can go a step further and ask them to provide anonymous comments about how they liked the search results.

Another approach would be to present a rating system near in the search results page which allows users to rate the quality of the search.

While these might provide you with some inputs about the quality of the search,they are very dependent on the user noticing the feedback/rating option. So another approach you can take is to see how many people actually select one of the videos on the search page and go to the video page instead of entering another search query. Another way of drilling down into this is to see the success of your search results would be to add a weight to each video (e.g. the top video would be the most weighted, the second video the second highest weighted and so on)

If you really want to drill down even more,you could do a measure of how much time the user spent watching the video before returning to the search results to try another option to see the level of satisfaction.

Now, I know a lot of this is theoretical and there are a lot of buts and if's but hey you asked for the solution of a theoretical question :)

  • To put in non-scientific terms, the happiness level that a user derives from a sorting algorithm is proportional to his enjoyment of watching some of the videos near the top of the search results.
    – JoJo
    Mar 8 '12 at 7:19
  • Maybe so but I wouldnt say all the videos is the key but one of the videos,since the user might be searching for a specific music video and unless he loves the song so much,I doubt he will listen to 10 different versions of it
    – Mervin
    Mar 8 '12 at 7:20
  • The more I think about this question the more I am drawn to the conclusion that your suggestion is the only way to do this. The drawback with AB tests is that they are anonymous and rely on computers being able to measure the results simply by counting one outcome over another. Happiness itself can't be measured in this way so the test subjects will need to actually do something to indicate their happiness. Even then this isn't ideal, because happiness are so subjective anyway even the most accurately determined 'popular' video might not fit the users expectations of popular.
    – JonW
    Mar 8 '12 at 12:46

It all depends on your goals and the type of the list you're creating.

Some lists, such as popular and latest, are constant to all users at the particular time period. Users don't care about your algorithm there because such lists are perceived as facts: a list of most recent content or the list of content with the most views/comments/likes/combination/etc in a time period. In such lists, it's completely up to you to decide which factors are more important for the rank: give more/less weight to the reactions you want to showcase (i.e. views, comments, likes, shares, etc.).

When you're dealing with lists that depend on user information, such as search results to a query or recommendations based on historical usage/profile/connections/etc, ranking is far more important. Yet, there are many ways to measure success. It can be the number of people who:

  • see only 1 of the first 5 items presented & don't check out the rest;
  • try all 5 top items presented and don't modify the query;
  • perform some action on the content page (save, rate, upvote, comment, etc);
  • don't visit the content page but perform an action from the page with the list.

The decision of what measure to use is completely up to you because it depends on your conversion funnel & business goals.

However, I would warn against relying only on self-reported surveying (e.g. asking "Was this helpful to you?") because they will depend greatly on the popularity of content and/or the feature. Take a look at this screenshot for the top review of Kindle Fire on Amazon:

enter image description here

In 4 months of its existence (as of March 8, 2012), 22,108 people of the millions who have visited the page have voted on its quality. If that's not striking, here's the top review for Fight Club (the book)

enter image description here

In the 9.5 years (as of March 8, 2012), only 398 people of hundreds of thousands have voted on its quality. That's very low.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.