- Structure the search engine report around relative strengths/weakness, if you can.
- Otherwise, help them raise their search game with a UX suggestions section to make search easier - i.e. if the engine is dismal on all counts.
"I am writing a report for a client
that tries to describe why their
search engine is failing users."
You must reduce the risk of a report that shows that the engine is failing users, but that does not show why. The problem is that if the engine isn't delivering accurate relevant results at all, saying so and saying so in detail is not a lot of use to the person who commissioned the report - even if poor selectivity is 'why' it is failing users.
- If your preliminary research shows that the search does serve some users well in some circumstances, then part 3 of your report will be built around the contrasts. For example it might be good for people who are looking for entry requirements for full time degree level courses in mainstream subjects identified using formal rather than colloquial terms. This would be its 'sweet spot' - and it might be very good indeed at that. But that might be a small percentage of visitors.
Tag and sort the user complaints, so that patterns of related complaints emerge.
You'll at this stage have evidence for where the search engine is relatively weak - e.g. (time) doesn't cater to people looking for evening courses or correspondence courses; (level) doesn't distinguish between courses offering recognized vocational courses and courses offering certificates; (synonyms) won't find 'mycology' if you look for 'fungi culture' or 'mushroom growing'; (course content) tells you what the entry requirements are, but not what is covered in the course.
"How do I come up with a way of
objectively measuring search accuracy
To convert that to measurement, it is quite OK to use a subjective assessment of whether a search (a user search or a made up one) was successful or not. It's the cumulative effect of multiple search attempts that yields valuable insight and makes the exercise sufficiently objective. Give graphs contrasting search success when in the sweet spot for each aspect versus when not. Overall success rates aren't useful for guiding change, but relative success rates are.
- If search is not serving any users well, you have a much tougher time. You have to do more detective work to find out why good results aren't being pulled up to the top of the list. Saying they aren't isn't enough. This is beyond pure UX work, and if you are in this particular extreme case you would have to offer something else for your report to add value.
"is there anything else you think I
should be including in my report?"
Search UX Recommendations
This is for when the existing search engine and UX is not serving any users well:
Rather than dive into the code for search and its metrics, your report can suggest a UX approach tailored to the domain that makes it easier for developers to write selective search code - but that work is still up to them. These are possible examples of the kind of thing you could say.
- One common problem is overly-similar results swamping the top spots in a list. (You do not want all 20 documents about Pharmacology in Bristol at the top - if the search is a general search for Pharmacology). So you propose an interface for telling users one result is actually a collection of related results AND for exploring that collection.
- You can propose an interface that presents synonyms that will be used in a given search, which the user can selectively disable.
- If it's a faceted search you may be able to suggest more appropriate facets, based on your analysis of requests actually made and documents that should have been returned.
In each case you back up your recommendation with evidence.
Other bits and bobs to consider
- Are there pdf docs in the store that are scanned from printed material? Have they been OCR'd? Try searching for some text you know is there.
- Is relevant information actually in the database? Try to hunt down a course in the database that includes fungi culture. Compare results with typing "mycology courses" into Google.
- How does google site-search compare with their custom search?