expert review of search engine - how to structure the report

Question

I am writing a report for a client that tries to describe why their search engine is failing users.

I am structuring the report like this:

section 1)expert review of interface. Will include heuristics such as placement and size of search box on screen, clarity and readability of results etc. This section will allow me to qualify what problems with the interface there are

section 2)Analysis of search logs: Google Analytics will tell me what the most frequent search terms are and if there are any patterns eg simple keywords or natural language queries

section 3)Testing search engine with a number of common search terms (n) to discover how bad/good the search engine is at delivering relevant, accurate results. This is tricky because there is a degree of subjectivity determining what is relevant or irrelevant; there will be different personas using the system for which relevance will vary.

question: section one and two are easy, but section 3 is a little bit more tricky. How do I come up with a way of objectively measuring search accuracy and relevance? Arbitrarily?

question: is there anything else you think I should be including in my report?

thx in advance.

Answer 1

For the third one you should come up with use cases that have a clear objective. Then create a few personas that try and execute that objective.

For example, someone wants to know the score of a football(soccer) game. How would each persona search and what would they get, and how much digging would it take.

Or run it through some usability testing. AppSumo even has a deal today for http://www.usabilitytesting.com

As far as what you should add, I would recommend a heat map, and any stats of a comparable site (to show as a sort of benchmark. i.e. Your bounce rate is 45%, this other site's bounce rate is 33%)

Answer 2

Relevance and accuracy are best measured by measuring user satisfaction.

One way to measure user satisfaction (other than usability testing) is to look at the business' results. Although conflating business results with user satisfaction is a slippery slope, using the chance at more money as leverage in an argument when trying to convince businesses can be quite effective. Find out how well users are converting by analysing SEO funnels (for instance, in Google Analytics) or other performance metrics the business has in place (such as number of sales from viewing search results).

For instance, if this is a hotel search site, find out how many people are searching for something, clicking a result, and booking a room. That will give you a number that you can compare to the business goals and draw conclusions from. Such as: the search results aren't very good because people clearly aren't booking as many rooms as you want them to.

It's a warped way of looking at the world, but it's often the one that matters to your client.

Answer 3

Summary:

• Structure the search engine report around relative strengths/weakness, if you can.
• Otherwise, help them raise their search game with a UX suggestions section to make search easier - i.e. if the engine is dismal on all counts.

---

"I am writing a report for a client that tries to describe why their search engine is failing users."

You must reduce the risk of a report that shows that the engine is failing users, but that does not show why. The problem is that if the engine isn't delivering accurate relevant results at all, saying so and saying so in detail is not a lot of use to the person who commissioned the report - even if poor selectivity is 'why' it is failing users.

• If your preliminary research shows that the search does serve some users well in some circumstances, then part 3 of your report will be built around the contrasts. For example it might be good for people who are looking for entry requirements for full time degree level courses in mainstream subjects identified using formal rather than colloquial terms. This would be its 'sweet spot' - and it might be very good indeed at that. But that might be a small percentage of visitors.

Tag and sort the user complaints, so that patterns of related complaints emerge.

You'll at this stage have evidence for where the search engine is relatively weak - e.g. (time) doesn't cater to people looking for evening courses or correspondence courses; (level) doesn't distinguish between courses offering recognized vocational courses and courses offering certificates; (synonyms) won't find 'mycology' if you look for 'fungi culture' or 'mushroom growing'; (course content) tells you what the entry requirements are, but not what is covered in the course.

"How do I come up with a way of objectively measuring search accuracy and relevance?"

To convert that to measurement, it is quite OK to use a subjective assessment of whether a search (a user search or a made up one) was successful or not. It's the cumulative effect of multiple search attempts that yields valuable insight and makes the exercise sufficiently objective. Give graphs contrasting search success when in the sweet spot for each aspect versus when not. Overall success rates aren't useful for guiding change, but relative success rates are.

• If search is not serving any users well, you have a much tougher time. You have to do more detective work to find out why good results aren't being pulled up to the top of the list. Saying they aren't isn't enough. This is beyond pure UX work, and if you are in this particular extreme case you would have to offer something else for your report to add value.

"is there anything else you think I should be including in my report?"

Search UX Recommendations

This is for when the existing search engine and UX is not serving any users well:

Rather than dive into the code for search and its metrics, your report can suggest a UX approach tailored to the domain that makes it easier for developers to write selective search code - but that work is still up to them. These are possible examples of the kind of thing you could say.

• One common problem is overly-similar results swamping the top spots in a list. (You do not want all 20 documents about Pharmacology in Bristol at the top - if the search is a general search for Pharmacology). So you propose an interface for telling users one result is actually a collection of related results AND for exploring that collection.
• You can propose an interface that presents synonyms that will be used in a given search, which the user can selectively disable.
• If it's a faceted search you may be able to suggest more appropriate facets, based on your analysis of requests actually made and documents that should have been returned.

In each case you back up your recommendation with evidence.

---

Other bits and bobs to consider

• Are there pdf docs in the store that are scanned from printed material? Have they been OCR'd? Try searching for some text you know is there.
• Is relevant information actually in the database? Try to hunt down a course in the database that includes fungi culture. Compare results with typing "mycology courses" into Google.
• How does google site-search compare with their custom search?

Answer 4

I don't know much about the nature of the search tasks, the amount of data and your user's goals, so I'll just focus my answer on general statistics to quantify relevance and accuracy. Here I assume results can be classified as relevant versus irrelevant, with nothing in between.

One approach for measuring (subjective) relevance (quite commonly used in domains like linguistics research) is to specify a coding scheme and have multiple people classify the search results. Some results will clearly be classified as relevant, others as irrelevant and others as relevant to some people but irrelevant to others. To get insight into the degree of subjectivity involved in the task, there are various statistical measurements for inter-rater reliability available (overview of different methods).

For accuracy you'd need to determine both how many irrelevant items were wrongly presented as results, and how many relevant items were wrongly left out. Simple accuracy: good classifications divided by total number of classifications. Sometimes false positives and false negatives are not equally bad, then you'd resort to specificity/sensitivity, positive/negative predictive value, etc.

However, if you don't really need to quantify your findings, you can focus on the subjective experience and user satisfaction and give a few examples of results that are clearly irrelevant to show what needs to be improved.

Added:

Louis Rosenfeld recently gave a workshop on Site Search Analytics at UX Lisbon (slides on slideshare, the part on measuring relevance and precision starts at slide 21). For relevance he measures the location of the 'best' result in the list (distance from the top). For precision he uses a rating scale with the categories relevant/near/misplaced/irrelevant, resulting in different scores for different permissiveness levels (strict/loose permissive).

Answer 5

In the absence of usability testing, I would recommend doing a cognitive walkthrough. Ideally you would do a walkthrough for the most important scenario for each of your personas.

For further information with examples, I can recommend reading the 4 questions to ask in a cognitive walkthrough at UserFocus.