This is probably wishful thinking but here goes...

Now obviously I don't expect some magical app to crawl a website, summarise each page and detail all the assets within each page (if only!) however is there a crawler of some sort that will trawl a given website and produce a full set of links and respective hierarchy would be invaluable. That would ensure that no pages get missed out as they invariably would through a purely manual inventory.

From there I would be able to see the existing site hierarchies and taxonomy decisions that have been made when building the site, and could use this as the basis for creating a detailed website content inventory.

Whether this be an online application, script or even a useful manual technique other than just manually sitting at the site and following links would be great.

1 Answer 1


Website Auditor is a tool that crawls your site and delivers a nested hierarchy, complete with meta info, broken link information, error codes, etc. We've found it VERY useful for the exact purposes that you list.

Alternatively, I've used Site Sucker for Mac (which is free) to download a local copy of an entire site, and used the log file as my content inventory. However, it's not as useful because it doesn't generate a nested view nor any of the niceties previously mentioned for Website Auditor, so it's definitely a "poor man's tool".

Not the answer you're looking for? Browse other questions tagged or ask your own question.