What is the threshold of trust for search?

Question

Assuming that you have to find what you are looking for x % of the time in order to trust a search service and become a habitual user of said service:

What number is x?

Links to research papers would be sincerely appreciated.

E.g. is there any officially available source for how often people find what they are looking for using google.com and related services? Are there regions (e.g. geographical) where findability is lower, and is it documented how this decrease affects customer loyalty?

Also, is there a documented region between y % and z % where trust is accelerating, and every additional improvement has substantial ROI (defined as increase in recurring use of the service)?

Finally, are there any enterprise search vendors that have stated how often their clients' employees should expect to find what they are looking for? E.g., have anyone heard sales people from Microsoft pitch such figures for SharePoint?

Background for the question: I have worked on enterprise search solutions for 12 years. Especially for "out of the box" SharePoint deployments, my takeaway is that many employees don't use the internal search for lack of trust, since findability is somewhere below 25%, sometimes even reaching the single digits (in stark contrast to the experience they have using google.com). To encourage owners to invest further, I assume it would be effective to show them where they currently are on the graph below, and at what point they can expect their employees to start trusting the service.

In this graph, "SharePoint" is understood as the product's default use case; "a document management and storage system" [W], where there is little or no editorial maintenance. "Intranet" is understood as the corporate site for editorial publication of internally relevant articles and tools.

One client states as a success criteria that "60% of the time, people should find what they look for". However, they do not have any data to back up that this is enough for people to start trusting the search. In my previous experience, if you can provide a findability of 75% the users return to the service, and vice versa, if it is lower that 30% then they shy away. But my sample of users and services are low, and I wouldn't refer to it as actual research. Also, the data points are too few to be of much value - you don't really need any data to infer that 75% is quite good and 30% is really bad.

Answer 1

From Christopher D. Manning; Prabhakar Raghavan & Hinrich Schütze (2009). "Chapter 8: Evaluation in information retrieval" (PDF). Part of Introduction to Information Retrieval.

User happiness is elusive to measure, and this is part of why the standard methodology uses the proxy of relevance of search results. The standard direct way to get at user satisfaction is to run user studies, where people engage in tasks, and usually various metrics are measured, the participants are observed, and ethnographic interview techniques are used to get qualitative information on satisfaction.

A number of quantitative measures to assess the quality of search engine results exist. This is the domain of information retrieval.

Evaluation measures (information retrieval) on Wikipedia give a good overview of existing measures. It doesn't include any benchmark that would allow you to compare your performance to successful search engines.

TREC (Text REtrieval Conference) publish evaluation results (password protected). They have a disclaimer regarding use as product testing, marketing and advertisement, but it might not cover internal use of results for advocacy to invest in more in search solutions.

Answer 2

What is your users goal? Is it to find a specific thing?

I design enterprise software too and the goal is rarely to wade through search results.

I recommend reading Search Patterns: Design for Discovery by Peter Moville & Jeffery Callender, and Ambient Findability by Peter Morville.

Designing a search system is not an easy exercise and it will need to involve a lot of collaboration with the technology team to create solution that helps the user find the thing they are looking for, quickly and efficiently.

Answer 3

I don't know that there is a magic number here (because magic numbers always have some secret behind them that we don't know about).

I think that you are looking at a number of factors here, and the combination of these factors allow you to assess what is a sensible starting point to test with your users.

• Accuracy of search terms/keyword: users may expect that if the term is quite unique or that they know exactly what they are looking for, then they expect a very high success rate (even if it is to say that there are 0 matches)
• Breadth of search parameters/scope: users may expect that if they are search through a very large repository that there is a higher chance of returning something that they are looking for, especially when they are certain of looking in the right space
• Number of expected results: users may expect that there will be many results returned, that at least one of more results will be returned
• Sophistication of algorithm: users may expect that a more complex or powerful search engine (e.g. Google) will be more likely to return results that they are looking for

I may have missed some, but hopefully these are the main ones.

So when you combine these factors and take into account that some users may also expect the opposite (due to less than satisfactory past experiences), you can see how there isn't a magic number you can associate with % success unless you understand what the expectations are.

Answer 4

I highly doubt that you can find any resource describing how many users find what they were looking for, because if I remember correctly, the search engines themselves don't know. They just take an educated guess from the info they currently know. Google has you move to a page and then redirects you to the page you wanted to see. If you press the back button, that page will log how long you took on site it redirected you to, and then redirect you back to the search results. There are probably other checks, but the above should point out what I'm trying to say clearly.

One could argue that if you never return to the results page, you could have found what you were looking for or on a referred link. But then again, it's hard to know what really happened if you don't ask the user or see what they did. In my case I could have run out of internet (when running on metered internet for those who didn't know).

As for your x number. That is a lot more complex. There are psychological factors that can come into play here, before even performance factors become relevant. The fact that a friend encouraged you to use a certain tool will usually lead you to using that tool more times than you think. I started using Google because a friend suggested it to me. It was the only one I knew at the time. But when I found out that there were others like, for example Bing, I still had a hard time trusting that these were as reliable as Google. I'm pretty sure if I had started with Bing I would have had the same "discomfort" towards Google. See many things can affect a person when it comes to trusting something like a search engine.

I know your question was probably hypothetical, but I just wanted to point that out.