It all depends on your goals and the type of the list you're creating.
Some lists, such as popular and latest, are constant to all users at the particular time period. Users don't care about your algorithm there because such lists are perceived as facts: a list of most recent content or the list of content with the most views/comments/likes/combination/etc in a time period. In such lists, it's completely up to you to decide which factors are more important for the rank: give more/less weight to the reactions you want to showcase (i.e. views, comments, likes, shares, etc.).
When you're dealing with lists that depend on user information, such as search results to a query or recommendations based on historical usage/profile/connections/etc, ranking is far more important. Yet, there are many ways to measure success. It can be the number of people who:
- see only 1 of the first 5 items presented & don't check out the rest;
- try all 5 top items presented and don't modify the query;
- perform some action on the content page (save, rate, upvote, comment, etc);
- don't visit the content page but perform an action from the page with the list.
The decision of what measure to use is completely up to you because it depends on your conversion funnel & business goals.
However, I would warn against relying only on self-reported surveying (e.g. asking "Was this helpful to you?") because they will depend greatly on the popularity of content and/or the feature. Take a look at this screenshot for the top review of Kindle Fire on Amazon:
In 4 months of its existence (as of March 8, 2012), 22,108 people of the millions who have visited the page have voted on its quality. If that's not striking, here's the top review for Fight Club (the book)
In the 9.5 years (as of March 8, 2012), only 398 people of hundreds of thousands have voted on its quality. That's very low.