Quantitative methods for analysing vast Intranet IA

Question

I have been asked to analyse and better group pages under a section of a large government website. The number of pages runs into the thousands.

The problems I have is that metadata rules have not been followed by staff so its very hard to make sense of what's going on. I have no idea if the current organisation is optimal or repetitious and I have been charged with finding this out ASAP.

So I can't trust the metadata. So what else could I analyse? Page titles? Any http://en.wikipedia.org/wiki/Data_mining that you have used and have found to be particularly useful.

I'm not interested in qualitative research at this point.

EDIT This problem is concerned with analysing a page and assessing whether the associated meta data (page title, page description) matches the contents. There is then to be a comparison with other pages in the directory to ensure that: a)it is in the right place and b)it is not repeated elsewhere. We are not altering the IA for it is governed by strange intransigent business rules.

Edit 2 There are about 6000 pages so tasks have to be quantitative and automated

Answer 1

Firstly I don't have experience in this area of doing content analysis on such a large website, but I have some ideas which may help or at least give some inspiration as to how you can make them relevant for your particular situation.

Consider if qualitative analysis can also help

You mention specifically quantitave methods, but since you are also looking for

• correctness of metadata relative to content
• the optimal (or otherwise) nature of the structure
• the amount of repetition or redundance,

then you are kind of pushing into qualitative areas as well, so don't necessarily rule out looking into whether there are some qualitative heuristics that might be able to be applied albeit in some crude automated fashion.

Consider what's going to happen when you're done

Look at how the resulting data is actually going to be used in order to make improvements. It's all very well getting a bunch of numbers back but that may not really be helpful in terms of taking next steps - only in determing if next steps need to be taken, so try to ensure not just the how much, but the whats, wheres, hows, and also the priorities.

For example if 25% of pages have bad metadata, but actually they are among the 25% of the least useful pages in terms of visits and time spent per page, then that's arguably less important to deal with than if 25% of the most heavily used pages having bad metadata. (although of course the bad metadata may be the reason why they are not receiving attention.

Consider relevance of a bottom up approach

You could look into a bottom up analysis of the site's source content, trying to automate the building of a mapping from the source to the web pages - again in order to determine relevance, repetition and redundancy. Combine this with page view data about how many visits or how much time is spent on those pages.

It's worth investigating how the content has been produced and by whom, and seeing whether after initial research there could be a pattern whereby the likelihood of bad metadata or repetitious content may correspond to

• a particular person preparing or submitting content
• a particular method of preparing or submitting content (including different software)
• a time period in which content was prepared or submitted
• other dependencies on the processes used

Consider data mining the use of the content as well as the content itself

One way of determining whether the organisation of content is working is by looking at the user's interaction and navigation with the page content

This would require some advanced use of google analytics or something similar. You may or may not have analytics already, (maybe not since it is an intranet and not necessarily already indexed - and maybe because I'd hope a site that size has something already in place).

If you do have any information on search results, page hits, time spent per page and the like, then you might be able to determine whether the interaction is efficient from the user's perspective:

For example, off the top of my head here's a couple of possible metrics that might give some clues:

Looking at efficiency of search results ...

• the most popular search terms
• the ranking of pages as a result of that search
• the number of visits to the pages in that list of results
• and the actual amount of time spent looking at each of those pages

...it may be possible to determine whether the pages that people are finding the most useful are not actually the ones that are appearing highest up in the rankings - thus indicating that the associated metadata, page titles and other SEO information on the pages is doing its job properly (Or that the search tools are not up to scratch).

Looking at how hard the user has to work ...

• the pages which people spend the most time on
• navigation routes to those pages from original landing pages
• the paths into the destination pages compared with time spent on the transitory pages on route.

i.e. - analysing how people by-pass pages of little relevance before they finally land on a page where they spend significant time.

If you can match data this against original search terms and the wording of links, then you might be able to see that some pages may seem to be appearing useful to the user, or are appearing higher in search results but are actually of little relevance other than to provide a long winded route to a real page of interest.

Given a very good set of analytics, you could data mine quite a wide selection of metrics in order to look for patterns and problems. If worrying patterns start to emerge, investigate why

If you don't have any analytics in place, consider installing some, depending on whether the site is heavily used enough in order to return you useful data in the time you have available - although I suspect this might be unlikely.

---

Some of this is clearly touching on SEO and maybe someone here has more experience in application and interpretation of analytics that can help?

Answer 2

If the information isn't classified, I would use humans for this task. Specifically, Amazon Mechanical Turk. Some tasks pay workers as little as a penny. https://www.mturk.com/mturk/welcome

I agree that page analytics would be helpful. Pages which aren't being visited could correlate with inaccurate metadata.

People don't follow rules if there's a reward for breaking them (saving time) and no clear reward for following them. The problem will continue until they have a set of pre-created templates to choose from vs. the ability to enter whatever they want... if it's faster to do the right thing, they're more likely to do it. Ideally, there would be not only a formal content policy but periodic content review.

Answer 3

I've seen a technique called TEXTY for fast text scanning without even read it, at Ars Electronica in Linz some years ago. It might help you.

Its made by Jaume Nualart and was set up on Ars Media Archive for check specific keywords, numbers (years) and names. Each category got a dedicated color and the word was overprinted with a colored bar or dot. All other words - not replaced by color bars - will be erased. Finally they had a sheet with colored bars and got an quick overview of its meaning by colors, amount of colors and its positions. I think looking for right keywords is the key to success.

Furthermore you can process some more visualisation techniques based on this. Check this link of Visualization Showcase of Ars Media Archive

The TEXTY technique used on former Ars jury statements and the team could show, how statements were shifting over years from very techy words in the begining to a closed art hemisphere by linking to artist names heavily in recent years.

Image from http://vis.mediaartresearch.at/webarchive/public/view/mid:44