Assuming that you have to find what you are looking for x % of the time in order to trust a search service and become a habitual user of said service:

What number is x?

Links to research papers would be sincerely appreciated.

E.g. is there any officially available source for how often people find what they are looking for using google.com and related services? Are there regions (e.g. geographical) where findability is lower, and is it documented how this decrease affects customer loyalty?

Also, is there a documented region between y % and z % where trust is accelerating, and every additional improvement has substantial ROI (defined as increase in recurring use of the service)?

Finally, are there any enterprise search vendors that have stated how often their clients' employees should expect to find what they are looking for? E.g., have anyone heard sales people from Microsoft pitch such figures for SharePoint?

Background for the question: I have worked on enterprise search solutions for 12 years. Especially for "out of the box" SharePoint deployments, my takeaway is that many employees don't use the internal search for lack of trust, since findability is somewhere below 25%, sometimes even reaching the single digits (in stark contrast to the experience they have using google.com). To encourage owners to invest further, I assume it would be effective to show them where they currently are on the graph below, and at what point they can expect their employees to start trusting the service.

enter image description here

In this graph, "SharePoint" is understood as the product's default use case; "a document management and storage system" [W], where there is little or no editorial maintenance. "Intranet" is understood as the corporate site for editorial publication of internally relevant articles and tools.

One client states as a success criteria that "60% of the time, people should find what they look for". However, they to not have any data to back up that this enough for people to start trusting the search. In my previous experience, if you can provide findability of 75% the users return to the service, and vice versa, if it is lower that 30% then they shy away. But my sample of users and services are low, and I wouldn't refer to it as actual research. Also, the data points are too few to be of much value - you don't really need any data to infer that 75% is quite good and 30% is really bad.

  • Are you suggesting that you can somehow manipulate the accuracy of your search results? Or are you trying to figure out if you need to make your search work better? – Andrew Martin Jun 21 '17 at 14:03
  • @AndrewMartin For our enterprise search, we're trying to build the business case for ongoing improvements including editorial efforts. – bjornte Jun 21 '17 at 14:17
  • 1
    I'm not sure you're going to find research for that but it would be a fairly obvious connection to make: search product is judged by it's ability to provide accurate results - more accurate results = better product = greater loyalty – Andrew Martin Jun 21 '17 at 14:19
  • 1
    I'm curious about the word "trust" as used here. It seems to me one is looking for how "worthwhile" the search is, or the expectation of utility. As I use the word "trust", there is a sense of safety in the presence of vulnerability. I don't trust Google. I expect searching on Google to be useful. – cmm Nov 15 '17 at 13:38
  • 1
    I meant no criticism of your background. It was not obvious to me that you were not a native English speaker. English is a language which I born into, and yet I still find new words, meanings, and constructions to learn. I thought it to be more likely that "trust" was a word used in a specialized technical capacity, and that you would explain that special use to me. Thank you for your response. – cmm Nov 16 '17 at 22:30

From Christopher D. Manning; Prabhakar Raghavan & Hinrich Schütze (2009). "Chapter 8: Evaluation in information retrieval" (PDF). Part of Introduction to Information Retrieval.

User happiness is elusive to measure, and this is part of why the standard methodology uses the proxy of relevance of search results. The standard direct way to get at user satisfaction is to run user studies, where people engage in tasks, and usually various metrics are measured, the participants are observed, and ethnographic interview techniques are used to get qualitative information on satisfaction.

A number of quantitative measures to assess the quality of search engine results exist. This is the domain of information retrieval.

Evaluation measures (information retrieval) on Wikipedia give a good overview of existing measures. It doesn't include any benchmark that would allow you to compare your performance to successful search engines.

TREC (Text REtrieval Conference) publish evaluation results (password protected). They have a disclaimer regarding use as product testing, marketing and advertisement, but it might not cover internal use of results for advocacy to invest in more in search solutions.

  • Gave you the bounty and an upvote since you gave the best answer – but of course the question remains unanswered :-) – bjornte Nov 5 at 9:45

What is your users goal? Is it to find a specific thing?

I design enterprise software too and the goal is rarely to wade through search results.

I recommend reading Search Patterns:Design for discovery by Peter Moville & Jeffery Callender, and Ambient Findability by Peter Morville.

Designing a search system is not an easy exercise and it will need to involve a lot of collaboration with the technology team to create solution that helps the user find the thing they are looking for, quickly and efficiently.

  • 1
    Yes, the individual use cases are mostly to find a single item (The type of item must obviously be disambiguated – hence see my older posts about e.g. group results). The only frequent exception is when employees want to see a collection of other employees, but of course this is also represented as a single item, the org chart. Agree that wading through results is not desirable. Good book tips, I have read both. And yes, I'm on the same team as the devs. – bjornte Jun 22 '17 at 13:12

I don't know that there is a magic number here (because magic numbers always have some secret behind them that we don't know about).

I think that you are looking at a number of factors here, and the combination of these factors allow you to assess what is a sensible starting point to test with your users.

  • Accuracy of search terms/keyword: users may expect that if the term is quite unique or that they know exactly what they are looking for, then they expect a very high success rate (even if it is to say that there are 0 matches)
  • Breadth of search parameters/scope: users may expect that if they are search through a very large repository that there is a higher chance of returning something that they are looking for, especially when they are certain of looking in the right space
  • Number of expected results: users may expect that there will be many results returned, that at least one of more results will be returned
  • Sophistication of algorithm: users may expect that a more complex or powerful search engine (e.g. Google) will be more likely to return results that they are looking for

I may have missed some, but hopefully these are the main ones.

So when you combine these factors and take into account that some users may also expect the opposite (due to less than satisfactory past experiences), you can see how there isn't a magic number you can associate with % success unless you understand what the expectations are.

I highly doubt that you can find any resource describing how many users find what they were looking for, because if I remember correctly, the search engines themselves don't know. They just take an educated guess from the info they currently know. Google has you move to a page and then redirects you to the page you wanted to see. If you press the back button, that page will log how long you took on site it redirected you to, and then redirect you back to the search results. There are probably other checks, but the above should point out what I'm trying to say clearly.

One could argue that if you never return to the results page, you could have found what you were looking for or on a referred link. But then again, it's hard to know what really happened if you don't ask the user or see what they did. In my case I could have run out of internet (when running on metered internet for those who didn't know).

As for your x number. That is a lot more complex. There are psychological factors that can come into play here, before even performance factors become relevant. The fact that a friend encouraged you to use a certain tool will usually lead you to using that tool more times than you think. I started using Google because a friend suggested it to me. It was the only one I knew at the time. But when I found out that there were others like, for example Bing, I still had a hard time trusting that these were as reliable as Google. I'm pretty sure if I had started with Bing I would have had the same "discomfort" towards Google. See many things can affect a person when it comes to trusting something like a search engine.

I know your question was probably hypothetical, but I just wanted to point that out.

  • 1
    Thanks for your reply. I agree that you probably can’t arrive at the answer only by relying on quantitative metrics, but I fully believe a qualitative study is feasible. In fact, I have done a number of them myself, so I know several ways of measuring findability as experienced by the end user, but I haven’t had the opportunity to study users as they approach the described threshold. – bjornte Nov 22 '17 at 12:07
  • @bjornte are you planning on trying a test now? Is this why you asked the question? – Kitanga Nday Nov 22 '17 at 12:55
  • @bjornte and if you are testing now, why not just test without any threshhold and look for the value range with the most users. That of course requires you to somehow get to that threshold. That now is your domain not mine :D – Kitanga Nday Nov 22 '17 at 12:57

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.