The influence of ratings on search result ranking

Question

Our product provides a search functionality. Search results are ordered by rank.

We also provide the possibility to rate the different search results: a user can (optionally) give each result a rating from 1 to 5 stars.

These ratings have no influence on the ranking of the search results. This was the recommendation of one expert we asked and it seemed very plausible to me, because rating and ranking are two different things.

E.g. sometimes I need a report or document which I don't like (and which I gave a bad rating) but which I have to use anyway.

That's the theory so far, now in practice user feedback shows, that people are expecting that their ratings have in influence on the search result order. Several users have asked why their top rated results are not put higher on the search results list.

Now the recommendation seemed quite good, but I don't want to fight against the users expectations either.

So my questions are:

1. Is it a good practice to strictly distinguish between rating and ranking?
2. Is there a way to fulfill this requirement and at the same time meet the users expectations?

Answer 1

You basically need to present a bunch of items and you have more than one metric to sort them by. Most websites who offer lists of items have this figured out well. You either present a sortable table where the rows are your items and the columns are the metrics, or do this via filters or tabs or any other navigation solution. Just make sure you provide an indication of all the relevant metrics in each view.

Google Play (note the open dropdown):

Amazon (same here):

Answer 2

It really depends on what your search results (and your ratings) represent.

Your results should always be presented based on rank, but (at least in some situations), content rating may be reflected in an item's rank for a given result.

What goes in to determining the best match is highly dependent on context and content of the things being searched, and especially on the context in which items are rated.

If you're searching in a content database (e.g. an image library) and you enter the term "garden" and get 200 results, all matching the term in the image's title or description, I'd expect to see higher-rated images that still meet the search criteria as being weighted higher than equivalently-ranked images that have been rated lower. In that case, I would expect to have my rating be incorporated (appropriately weighted) into the search ranking algorithm. In general pages/images/items rated more highly are more likely to be the page users are looking for if they also appear in search results for the given keyword based on content.

Alternatively, if what you're developing is a search engine first-and-foremost (e.g. if you're Google) and the ratings are provided in the search results interface itself, I would read a rating being provided as a rating of that content in the context of my search term, not a rating of the content itself. As such, I would expect my 5-star rating of a given result to affect identical or very similar searches in the future. I would not expect that rating to be stored as a value of the content (the same result, in this design, could be rated by the same individual as both a 1-star and a 5-star result given different search keywords).

In either scenario, the rating shouldn't be the only factor in the ranking of results, or even the main one, but it should be a factor.