Best way to setup search for large amounts of PDFs etc

Question

My first post here and hoping to get some useful insight and feedback. I'm working on a redesign project (UX & IA) for a local gov't client with a large amount of PDF documents. One of the project requirements is adding search to facilitate users finding these PDFs easier. Here are a few steps I'm already taking:

1. Identifying and transitioning key PDFs to HTML pages for better search indexing. The balance of the other PDFs I'm not bothering with since G.A data tells me there's little to no traffic therefore no user value to warrant HTML.

2. PDFs currently live in a document repository as part of the global nav. Again GA tells me there's not much traffic to this section so I'm moving the most important PDFs from this category and filtering them into contextually relevant parent categories across the site for better user discovery and findability

3. PDFs are not sorted in any logical way currently (ie: alpha, numeric, date/year etc..) I plan on creating groupings of similar PDFs using clear understandble labelling using contextual vocabulary.

My question is, to facilitate the search function returning the best/accurate results I'm assuming we'll need to set up a logical meta tag system in the CMS.

A. Does it make sense to first filter out any R.O.T (site wide) so only the most relevant content is surfaced up in a users search query?

B. Do my above 3 steps make sense and am I missing anything crucial here?

Thanks in advance -gary

Answer 1

It's difficult to answer this question because the answer depends a lot on how much time/resources you have, and what kind of search you are looking for.

For example, if you are indexing court documents where a search must retrieve every matching record, no matter how small or trivial, then you cannot really remove results a priori. On the other hand, if the documents are voluminous and you don't want to overwhelm users, then some mixture of curating, indexing and scoring makes sense.

So this is a question that does not have a definitive answer and is not a great match for this site.

That said, if it helps:

• Unless you need to display the documents in HTML, there is no need to migrated PDF to HTML. If you're doing this for indexing reasons as you suggest in the question, then you are better off either moving to text files with much less tag overhead for search, or to a database if you have resources to do that.

• (meta) tags are a great way to help set up content for search. They are great because you don't have to prescribe a taxonomy or any kind of orthogonal classification, and they can be migrated in lots of different directions later on. For example, just tagging by document creation date, name of directory, etc can be helpful.

• When setting search, incorporating user feedback is critical. Ensuring that your search index has capability to measure user clicks and automatically upgrade relevance scores is very important.

Answer 2

One suggestion would be to group them (like you suggested) and make sure they have clear titles/names. Perhaps even thinking of them like the way some academic articles/journals are set up - they have authors, title, date, summary - basic information but the rest of the information is in a PDF.

Figure out what people might want to search about the PDFs and go from there. Good luck!