Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

I have been asked to analyse and better group pages under a section of a large government website. The number of pages runs into the thousands.

The problems I have is that metadata rules have not been followed by staff so its very hard to make sense of what's going on. I have no idea if the current organisation is optimal or repetitious and I have been charged with finding this out ASAP.

So I can't trust the metadata. So what else could I analyse? Page titles? Any http://en.wikipedia.org/wiki/Data_mining that you have used and have found to be particularly useful.

I'm not interested in qualitative research at this point.

EDIT This problem is concerned with analysing a page and assessing whether the associated meta data (page title, page description) matches the contents. There is then to be a comparison with other pages in the directory to ensure that: a)it is in the right place and b)it is not repeated elsewhere. We are not altering the IA for it is governed by strange intransigent business rules.

Edit 2 There are about 6000 pages so tasks have to be quantitative and automated

share|improve this question
    
Do pages point to each other? If so you can model them as a graph and find clusters. What are the practical implications of grouping? Is it for browsing/finding pages? –  Assaf Lavie Oct 30 '11 at 18:35

3 Answers 3

up vote 6 down vote accepted
+200

Firstly I don't have experience in this area of doing content analysis on such a large website, but I have some ideas which may help or at least give some inspiration as to how you can make them relevant for your particular situation.

Consider if qualitative analysis can also help

You mention specifically quantitave methods, but since you are also looking for

  • correctness of metadata relative to content
  • the optimal (or otherwise) nature of the structure
  • the amount of repetition or redundance,

then you are kind of pushing into qualitative areas as well, so don't necessarily rule out looking into whether there are some qualitative heuristics that might be able to be applied albeit in some crude automated fashion.

Consider what's going to happen when you're done

Look at how the resulting data is actually going to be used in order to make improvements. It's all very well getting a bunch of numbers back but that may not really be helpful in terms of taking next steps - only in determing if next steps need to be taken, so try to ensure not just the how much, but the whats, wheres, hows, and also the priorities.

For example if 25% of pages have bad metadata, but actually they are among the 25% of the least useful pages in terms of visits and time spent per page, then that's arguably less important to deal with than if 25% of the most heavily used pages having bad metadata. (although of course the bad metadata may be the reason why they are not receiving attention.

Consider relevance of a bottom up approach

You could look into a bottom up analysis of the site's source content, trying to automate the building of a mapping from the source to the web pages - again in order to determine relevance, repetition and redundancy. Combine this with page view data about how many visits or how much time is spent on those pages.

It's worth investigating how the content has been produced and by whom, and seeing whether after initial research there could be a pattern whereby the likelihood of bad metadata or repetitious content may correspond to

  • a particular person preparing or submitting content
  • a particular method of preparing or submitting content (including different software)
  • a time period in which content was prepared or submitted
  • other dependencies on the processes used

Consider data mining the use of the content as well as the content itself

One way of determining whether the organisation of content is working is by looking at the user's interaction and navigation with the page content

This would require some advanced use of google analytics or something similar. You may or may not have analytics already, (maybe not since it is an intranet and not necessarily already indexed - and maybe because I'd hope a site that size has something already in place).

If you do have any information on search results, page hits, time spent per page and the like, then you might be able to determine whether the interaction is efficient from the user's perspective:

For example, off the top of my head here's a couple of possible metrics that might give some clues:

Looking at efficiency of search results ...

  • the most popular search terms
  • the ranking of pages as a result of that search
  • the number of visits to the pages in that list of results
  • and the actual amount of time spent looking at each of those pages

...it may be possible to determine whether the pages that people are finding the most useful are not actually the ones that are appearing highest up in the rankings - thus indicating that the associated metadata, page titles and other SEO information on the pages is doing its job properly (Or that the search tools are not up to scratch).

Looking at how hard the user has to work ...

  • the pages which people spend the most time on
  • navigation routes to those pages from original landing pages
  • the paths into the destination pages compared with time spent on the transitory pages on route.

i.e. - analysing how people by-pass pages of little relevance before they finally land on a page where they spend significant time.

If you can match data this against original search terms and the wording of links, then you might be able to see that some pages may seem to be appearing useful to the user, or are appearing higher in search results but are actually of little relevance other than to provide a long winded route to a real page of interest.

Given a very good set of analytics, you could data mine quite a wide selection of metrics in order to look for patterns and problems. If worrying patterns start to emerge, investigate why

If you don't have any analytics in place, consider installing some, depending on whether the site is heavily used enough in order to return you useful data in the time you have available - although I suspect this might be unlikely.


Some of this is clearly touching on SEO and maybe someone here has more experience in application and interpretation of analytics that can help?

share|improve this answer
    
Some really interesting points Roger and thanks for the time you've obviously spent providing such a detailed answer. Yes, I agree: I am already constructing heuristics to compare page contents with metadata quality. And yes, SEO is always a great tool for structuring the IA, but we're not altering the IA (you know as well as I do that government websites are kinda set in stone). The kicker with this problem is trying to find out which pages roughly say the same thing as others. Maybe once we have the page titles and metadata in place it will be a much easier process. I dont know. –  colmcq Oct 31 '11 at 10:13
    
well done big yin! you win my epic prize. Thank me laterz. –  colmcq Nov 1 '11 at 20:16

If the information isn't classified, I would use humans for this task. Specifically, Amazon Mechanical Turk. Some tasks pay workers as little as a penny. https://www.mturk.com/mturk/welcome

I agree that page analytics would be helpful. Pages which aren't being visited could correlate with inaccurate metadata.

People don't follow rules if there's a reward for breaking them (saving time) and no clear reward for following them. The problem will continue until they have a set of pre-created templates to choose from vs. the ability to enter whatever they want... if it's faster to do the right thing, they're more likely to do it. Ideally, there would be not only a formal content policy but periodic content review.

share|improve this answer
    
I think this is a helpful-ish answer but I did vote it down initially and can't get it back to zero so I've left at +1. I've edited my OP to specify the solution has to be automated (because of the sheer volume of analysis to be done). Your last paragraph is interesting for correct generation of meta data moving forward; tbh, humans shouldnt have to generate the metadata. –  colmcq Nov 2 '11 at 9:43

I've seen a technique called TEXTY for fast text scanning without even read it, at Ars Electronica in Linz some years ago. It might help you.

Its made by Jaume Nualart and was set up on Ars Media Archive for check specific keywords, numbers (years) and names. Each category got a dedicated color and the word was overprinted with a colored bar or dot. All other words - not replaced by color bars - will be erased. Finally they had a sheet with colored bars and got an quick overview of its meaning by colors, amount of colors and its positions. I think looking for right keywords is the key to success.

Furthermore you can process some more visualisation techniques based on this. Check this link of Visualization Showcase of Ars Media Archive

The TEXTY technique used on former Ars jury statements and the team could show, how statements were shifting over years from very techy words in the begining to a closed art hemisphere by linking to artist names heavily in recent years.

enter image description here Image from http://vis.mediaartresearch.at/webarchive/public/view/mid:44

share|improve this answer
    
my earlier comment seems to have vanished! this is an interesting post but in no way meets the criteria of the OP. I'm looking for quantitative methods, automated preferably, for analysing large IAs that have incomplete or inaccurate page metadata; specifically tools that look at page content and generate categorisation rules. –  colmcq Oct 30 '11 at 19:27
    
Okay, after reading your comments I think its more clear for me now. I showed you this project, because they had same problem: a huge archive with texts from over 20 years and no metadata at all. This TEXTY thing was their way for getting Metadata out of it. But its hasn't to be colors, can be numbers or IDs as well. They chosed colors for aesthetic showcase reasons. For me its not really clear, how you want to –  FrankL Oct 31 '11 at 22:18
    
Sorry, nervous finger - now part two: How could you work with that: Based on this ID combinations, which every page has, you could check and compare all pages statistically. –  FrankL Oct 31 '11 at 22:28
    
yes! and that's what I'm trying to figure out, but the 'how' is the hard part. I'd will say thanks for your input. Much appreciated :) –  colmcq Oct 31 '11 at 22:42
1  
... Assumed you know which keywords are relevant and have to be tracked. This isnt a tool were you enter your URL and get readily formated reports out, its a approach or a path I suggest. Because I know the guys who analysed the archive and they told me there isnt an out of a box solution for work like that. But its 2 years ago and Im not active in this area, may be there is a tool. (mmmh 5 minutes for a comment edit is too fast for me .( –  FrankL Oct 31 '11 at 22:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.