Fuzzy search by default?

Question

There's a search box, powered by Apache Lucene, that user enters terms into to find records that they need, (eg. They're likely to be searching for people, or addresses, or ID numbers, etc).

Example queries:

John Smith
John OR Bob
Name:John Country:Australia
Name:(John OR Bob)
Name:[Andy TO Charlie]
Name:"Mary Jane"

Lucene allows you to add a fuzzy modifier to terms, to find similar terms.

John~ //by default the fuzzy value is 0.5, lower value means more fuzzy.
John~0.1 //Might return James, Joan, Jon etc
John~0.8 //Might return Jon

I've been tasked to make all queries fuzzy by default, so for a simple query, when the user enters 'John Smith' he gets results for 'Jon Smith' etc.

The way to achieve this is to parse and change the query John Smith to John~ Smith~.

If the user needs to turn the fuzzy search off, they can specify the fuzzy term John~1 Smith~1.

The issue is, this gets a bit more complicated for the less standard queries.

For example should:

Name:(John OR Bob) evaluate to Name:(John~ OR Bob~)

John NOT Jon -> John~ NOT Jon~

?

The issue with the John NOT Jon query is that when Jon is fuzzified all those results are excluded, this query is likely to not return any results.

One option is to say 'Making results fuzzy by default isn't a good idea, you should just train the users on how to use the fuzzy modifier'.

Another option is to come up with a series of conditions for when a term should be fuzzyified and when it shouldn't.

Ie.

Fuzzy any standalone terms.
Don't fuzzy anything in square brackets.
Don't fuzzy anything in double quotes.
Don't fuzzy anything following a NOT.
Fuzzy stand alone terms in round brackets. 

The problem I see with this, is that it might make things confusing for the user, as to why sometimes a 'John' term returns 'Jon' and why other times it doesn't.

From the initial problem (we want to make basic searches fuzzy) we have actually a kind of complicated problem.

I'm wondering what the best set of rules or conditions here is, such that it's not needlessly complicated or hard to use for the user.

Answer 1

I think it comes down to your default search addressing the most common use cases first with fuzzy search on. Any edge cases could be satisfied by an "advanced search" function to avoid a cluttered interface.

I suggest to have next to the search field a checkbox labelled "Fuzzy search" checked by default. Users are now free to uncheck it and see if that leads to more accurate results. refreshing the results based on the checkbox state would be nice.

You could always give users more options to refine their search results by making more settings available in an "Advanced Search" drawer.

Answer 2

As you already told in the question, the problem with fuzzy search is that the user might not understand why some search results are being displayed. So here is my suggestion:

1. When the user types - John:
   • If there are several results matching the query, return the results
   • Otherwise, fallback to a fuzzy search
2. When the user types - "John and Mary": do the query, but don't fallback to a fuzzy search.

The problem with this is that I will need to fine tune when to fallback to a fuzzy search, and also have a way to ensure that results that match exactly the query will be displayed first.

Regarding using brackets, square brackets,... I would use double quotes. So the query

John "Country:Australia"

would mean: "give me all people that live in Australia and have a name that is similar to John".

Answer 3

The difficulty you will with the explicit 'fuzzy search' option will be how to clearly message/label it to your users in a way they will understand.

Instead, I would suggest using the fuzzy results and order them by strength - most accurate matches first, looser matches following shortly behind. For Example:

--

John Smith returned 5 Results:

1. John Smith
2. John Smith
3. Jon Smith
4. J Smith
5. John Smit

--

In addition to this, if a user wishes to do a strict search (non-fuzzy) it is fairly common to support the use of quotes ( " or ' ) which would disable the fuzzy matching for that search term. It also gives the ability to combine strict and fuzzy, for example John "Smith"

Also providing a set of sort options can help if you are dealing with a large pool of data. Options like 'Relevance' 'Alphabetical' 'Newest' etc. can add greater utility.