Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I'll soon need to be testing how well a recommendation engine is producing search results, per user. There will be some personalization inputs and I'd like to test how well the results are with varying levels of completion of the inputs.

I'm curious about any proven practices or resources.

share|improve this question
    
Welcome to the site @Scott! Please provide some more detail. (Who are your users? What will they hope to accomplish? Why?) At the moment your question is too broad to answer. –  3nafish Dec 12 '13 at 3:31
    
I agree, some clarification would be nice. Favorited it though, this question is highly relevant for the current project I'm working on –  magnus.westrom Dec 12 '13 at 8:12
    
There are established metrics for recommendation quality - serendipity, novelty, etc., as well as lots of published studies. Search for such publications e.g. on Google scholar, and you will see how the studies are done and how the metrics are calculated. –  Rumi P. Dec 16 '13 at 11:53

1 Answer 1

Plain old straight forward User Testing should reveal how effective your recommendations are. Just establish a set of criteria for threshold acceptability and run a few trials to see if your system reaches these limits.

The key here is trying to determine acceptable success criteria that directly relates to the value you're trying to add be incorporating this feature. If it were my test I'd want to know:

  1. What percentage of users noticed the recommendations

  2. What percentage of users followed the recommendation links/prompts

  3. Of the links/promts followed, where in the displayed list did they rank

You could get even more dense with your criteria (such as how long did it take the user to find and act upon a recommended link), but unless you're planning on testing a fairly large sample the value of this level of resolution is questionable.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.