Whats the best default for search result sorting?

I'm working on a news search, and I find that sorting by the number of keyword matches tends to produce stale results, but sorting by date tends to produce irrelevant results.

How do I strike a balance? I'd rather have a good default so users don't have to muck about the settings themselves.

-

TL;DR: Use a multi factor ranking system.

A good example to follow is the way that Google rank search results. We of course don't know the precise details of their ranking algorithm, but they have arguably done the most research on this and have the most success. What we do know for sure is that Google include a large number of factors and apply a weighting to each to give a final results ranking.

I'll try to give you a crude example of how it could be done:

Assign a value to each result based on its age.
the last 10 minutes = 100
the last hour = 80
the last 6 hours = 70
the last day = 60
the last week = 40
etc.

Assign a value to the keyword density
5 or more matches = 100
4 matches = 80
3 matches = 60
2 matches = 30
1 match = 10
0 matches = 0

Create a weighting matrix
Date vale = 8
Keyword density value = 4

Work out the rank value for each article
Multiply the value for each factor by its value in the weighting matrix for each article

An article from 1 day ago with 4 keywords would have a rank value of:
60 * 8 + 80 * 4 = 800

An article from 10 minutes ago with 3 keywords would have a rank value of:
100 * 8 + 60 * 4 = 1040

You would then show each result by it's computed rank value.

Some things to note here. You don't strictly speaking need the weighting matrix, but it makes it easier to tune the results - which you should do. Also, to get good results you would probable need to include more than two factors. You could for example assign weighting to the length of the article; or the publisher of the article. All these choices are really more of an art than a science, so you will need to play around with it a bit.

-
Thanks for spelling it out for me. In doing research, I found amix.dk/blog/post/19588 which explains how reddit sorts posts. After playing with it I've found a good algo for balancing freshness and matches. Thanks again for the pointers. – Hayk Saakian Jan 27 '13 at 4:25

As mentioned in other answers, there's not going to be a single factor you can sort by to ensure the relevant results are at the top. You'll need to combine multiple factors, this article on the Reddit ranking formula may point you in the right direction.

I would try using it with number of matching keywords (possibly weighted by title/body) and date. You'll have to tweak the numerical factors though as you presumably aren't as heavily focused on the new items as they are at Reddit.

-

How about giving more weighting to keyword matches in the title of the news article, and in the short description (if there is one). You could assign more points to the matches in the title, less for the short description, and even less for the body text.

Then, depending on the date of the news article, you could give additional points depending on it's recency.

Then, total up the points, and order by most points to least points.

Obviously this algorithm will need some tinkering with, but I think this would make a good starting point. Keep adjusting until it works.

-
I'm playing with this right now. Using matches as score with the reddit post algorithm. Why give more weight to titles? In some tests, I found that this promoted short articles that had keywords in the title over long articles that had keywords multiple times in the body. – Hayk Saakian Jan 27 '13 at 0:34
You've got a good point there. My thought was, if the keyword is in the title, most likely the whole article is about that keyword. If it's only in the body, then it might be just a passing reference. But, you're right, if the keyword shows up multiple times in the body, the article is most likely about that keyword. So if the title has any increased weighting, I think it should be very slight. – Rich Jan 27 '13 at 0:59