Rapid content audits, can this be done?

Question

Content audits although immensely dull are essential to understanding an existing websites content. Content audits for larger sites where you're capturing each page and analysing the content page by page is obviously very time consuming. I'm wondering if there is a "lean" approach to auditing/quick way to understand a websites content without having to go through the entirety of the site?

An example I thought of was creating a sitemap of the site. This will give you an indication of the shape and size of the site. In terms of capturing content you would only be capturing the pages you believe are unique content types. Rather than going through the whole site. I guess though in order to discover the content types you would still potentially have to click through the whole site but it would save time of not having to capture all the URL's and describe each page.

Answer 1

The statisticians use sample method for such tasks. All of your pages are general set. To make some inferences about all the set you need:

1. Create some assessment procedures.
2. Pick subset of general set. Paying more attention to important pages has sense.
3. Assess every item in subset using your procedure.
4. Create report.

This way could significantly shorten analysis time while results would be of acceptable quality.

Answer 2

I'd put the site into http://opensiteexplorer.org/ and go to the section that lists top pages (those pages most linked to on the domain).

This will give a solid idea of content that has performed well.

Further, augment this with an analysis of the site's Google Analytics page data... see which pages have historically received the most traffic.

Then:

1. Make a log of the top performing pages (content) on the site
2. Categorise these pages loosely (take into account any categories already in place on these pages by referring to the URL)
3. Determine which categories have the most high-performing pieces of content, and use this insight to dive in and trawl for more quality content within these categories.

You could plot the Opensiteexplorer and Google Analytics data against the sitemap to get a visualisation (Heat map) of the popularity (and thus to a large extent, value) of the site's content assets.

Answer 3

The purpose of doing a content audit is to assess the quality and effectiveness of your content against a set of goals--business requirements, customer tasks, performance metrics.

Ultimately, what you are asking is how to speed up something that requires human effort and judgment, but there are ways to automate the data gathering to speed up that cataloging part of the project and get you to the brain work faster. I recommend using a tool, such as ContentWRX Audit, from Content Insight, to create the inventory that you need to start your audit. It gives you a complete list of all the pages, images, media files, and documents on the site as well as the links in and out from every page, so you have a base to start from in terms of understanding the quantity of content and the structure. The tool also integrates Google analytics data, so you already have a step in the direction of figuring out what content to focus on.

To scope your actual audit effort, use that analytics data to look at your highest and lowest performing content (highest to figure out what's working and why; lowest to find out what needs to be revised or removed). You can also scope an audit by focusing just on the pages important for your highest-value user tasks--product evaluation through to buyflow, for example. The context for your audit also gives you some ways to limit the effort if you don't have a lot of time--a complete site redesign project is going to be more about looking for site architectural issues and content and interaction models. A rebranding project is going to be more focused on the actual writing and creative and will require a lot more time spent with the content itself.

Another tactic once you have the inventory is to "divide and conquer" the human effort surrounding the audit, for example, an Information Architect could review structure, a Content Strategist could look at content, a Business Analyst reviews functionality, etc.

In a nutshell--know your context and know your goals, and use tools to help you with the stuff the tools do well so you can spend your (and/or your team's) limited time doing the actual analysis.

Answer 4

The best option would seem to be combining your idea and Alexey Kolchenko's (and using micap's script idea where applicable):

1. Get the topology/structure/hierarchy of the site.
2. Do sampling from within each node of the topology (except nodes for individual pages, of course).

You will probably want to make the sample size for different nodes be proportional to how much content is at that node (i.e. larger samples from larger sections), but the point is that you will want at least something from each node.

Answer 5

(1) Create your sitemap: it would definitely give you a great understanding of how the content has been structured.

(2) Compare your sitemap with analytic data: prioritize and structure your sitemap based on your findings.

(3) Recommend any removal/modifications: by comparing the current content structure against your new modified content structure sitemap.

(4) Don't forget to use nice bright colors and a legend to help visualize your recommendations.

Answer 6

You can try the concept of Text mining for this type of work. Following are the steps can get you very good insights about the website without going through even a single page.

Step 1: Crawl all the pages of the website. if the Pages are less than you can set the web urls manually in the script and run the loop for all the urls and get their text

Step 2: If the website is big, try to build a crawler using Apache Nutch or take use of some online web app which can do it for you. Probably you may find some Python or R script which can help to get this done.

Step 3: Once you have text content of all the pages apply Text mining to perform-

a) Sentiment analysis for each page (Which can tell you whether the content on the page is positive/negative and by how much)

b) Unsupervised Topic modelling using LDA which can tell you this page has content about which topic

c) Feature based opinion mining to get the understanding on what feature like politics, sports what people are talking about.

Advantages of this approach:

1. You don't need to go through a single page for analysing the content
2. It can be applied to any website(if you are not crawling them every single day)
3. Single R or Python script can perform the task and its not that hard as it may sound (I have done the same in R for my company)

Answer 7

I'm going to assume you have complete access (but some of these techniques don't require it, and I'll give alternatives in case you don't have access).

Download the site via FTP (or use a wget script--command line) and you'll get a pretty good idea of structure. If it is a dynamic site (e.g. using PHP and a database) make sure to manually review at least one page of each type (each template) and look for things which require human discretion: styling, aesthetics, wording, weirdness at different (common) screensizes, if responsive.

Once you have a local copy it's very easy to search through all the files (or the database) so that if you want to know how many times the word 'foobar' occurs on the site you can use the command line with something like grep or something as simple as the find option in Sublime Text 2.

Use the 'audits' tab in Chrome developer tools (I'd also use PageSpeed or YSlow) on the live site (again if it is a dynamic site one instance of each 'template' should be representative).

Validate the code; the W3C has a new validator that looks at an entire site for a wide range of issues. A broken link checker (W3C has one also) might be good too.

Next, I'd look through webmaster tools and analytics. A post (an entire blog actually) that I think you'll find useful is Avinash Kaushik's Intro to Web Analytics. Step 6, which uses a keyword cloud, immediately came to mind and can be used without access to analytics.

Obviously, these steps won't replace a manual review but they will give you a really good idea where to look.

Oh, and one more little known trick, go to Google and search site:example.com, this gives you a fast way to see all the pages in Google's index.

Update: I recently started a new job (SEO) and we use the Screaming Frog SEO spider tool. You get more features if you buy a license but you get most of the functionality in the free version (and at least then you can try it out). It really helps with speed up audits of large sites.

Answer 8

I've done similar projects before, basically I use the sitemap.xml file (or create one with Integrity http://peacockmedia.co.uk/integrity/ ) and then run a script that converts the sitemap.xml file to an omnigraffle drawing (this is good https://github.com/jdkunesh/Sitemap-Wayfinding-Applescript-Generator--SWAG-- ).

Then I go through the drawing and prune the sitemap to make sense of it. That way I get a sense of the content and the relative focus of the website.