2

I'll soon need to be testing how well a recommendation engine is producing search results, per user. There will be some personalization inputs and I'd like to test how well the results are with varying levels of completion of the inputs.

I'm curious about any proven practices or resources.

3
  • Welcome to the site @Scott! Please provide some more detail. (Who are your users? What will they hope to accomplish? Why?) At the moment your question is too broad to answer. – Graham Herrli Dec 12 '13 at 3:31
  • I agree, some clarification would be nice. Favorited it though, this question is highly relevant for the current project I'm working on – magnus.westrom Dec 12 '13 at 8:12
  • There are established metrics for recommendation quality - serendipity, novelty, etc., as well as lots of published studies. Search for such publications e.g. on Google scholar, and you will see how the studies are done and how the metrics are calculated. – Rumi P. Dec 16 '13 at 11:53
2

Plain old straight forward User Testing should reveal how effective your recommendations are. Just establish a set of criteria for threshold acceptability and run a few trials to see if your system reaches these limits.

The key here is trying to determine acceptable success criteria that directly relates to the value you're trying to add be incorporating this feature. If it were my test I'd want to know:

  1. What percentage of users noticed the recommendations

  2. What percentage of users followed the recommendation links/prompts

  3. Of the links/promts followed, where in the displayed list did they rank

You could get even more dense with your criteria (such as how long did it take the user to find and act upon a recommended link), but unless you're planning on testing a fairly large sample the value of this level of resolution is questionable.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.