How to visualize similarity of one file to another?

Question

Problem

My file-system is a mess, over the years I've been accumulating way too many downloads I didn't care to uncompress and organize, code that was subject to endless copy+pastes and improvements and lots of images and documents that are basically the same file just with some minor changes/revisions.

I want to display the similarity of a file to any other file, as long as it is higher than the given threshold. I also want to do the same with folders.

Clarification: This is not about visualizing the differences between any 2 or more files, this is about how to represent the similarity of every file with all the other files (many to many).

---

Background

What I thought would be a good approach would be to have two kinds of perspectives: one that would show only files and another one that would show only files. Then I would display a circle (perhaps the circle dimension could be directly proportional to the file/folder size) for each node (file/folder) which I would connect to all the other similar nodes. The distance between the nodes would be inverse of the similarity between them, and the actions would be available through a context or side menu.

This idea has a couple of problems though:

1. if there are many nodes, either the circles would have an impossible small area, or the distance between them would be minuscule or the scrolling area would be annoyingly huge

2. if the node size was directly proportional to it's file size, then the above problem would be even more accentuated, or there would be some overlaps that would make the whole navigation unusable.

I was thinking of using something like this (but this one doesn't reflect the strength of the similarity) or like this (which has the problems I described above). Is there any design pattern that would fit here..?

I would like to do something like the Muffin Player:

But they have the advantage of being able to break their nodes into only 2 axis and they also have the depth dimension (although I'm not exactly sure about it's purpose).

Answer 1

As you mention a simple scheme alone would very likely run into information overload, with too many nodes overlapping one another. To my mind you need to take a combined approach, and likely start by allowing filtering of the search results, say to a given folder tree.

As for the combined approach I think you need to consider combining elements from the two following high-level areas.

Highlight the current file being compared.

Possibilities include:

• Color - Make it red, or green or white or ...
• Size - Make it the largest object
• Shape - Make it a circle, square or triangle, although I like the idea of a target
• Position - Make it central to the screen, or in the top left or right OR set it off in an explorer like view.

Differentiate all additional files

• Color
  • Simple scheme: Red files are different, Yellow files share minimal content, Blue files are share a significant portion of content and Green files are identical
  • Shading: lighter as the files vary, the more they vary the lighter they are from the current shade. Possibly changing to another color along the spectrum as above.
• Size - Smaller files share less in common with the currently selected file as objects that are farther away appear smaller.
• Shape: Maybe star shapes are close matches, maybe pointy shapes are further away and rounded shapes are closer. This one could get vary subjective.
• Location: files closer to the target more closely match it, could be combined with size above to give a familiar three dimensional feel. Like navigating nodes on a solar system chart in a console game.

Answer 2

If you're trying to visualize a file structure (a tree), and are trying to find ways to prune it, there are some things you can do.

The size of the blob needs to be associated with the size of the file.

The distance from the center could be associated with the distance from the root (C:)and on selection could be associated with the LCS (largest common sub-string) the longer the sub-string the closer the nodes should be.

The color could be associated with the file type association, media could be orange, executable blue etc.

Maybe you want to circle sets of related files (by folder) and show the size of a that. I know frequently those little files can add up and take up a ton of space, but the reason why you don't prune them is because you don't know where they're coming from.

if your differences in size are huge, allow the user to choose for example a log scale algorithm. EX: xkcd.com/482/

However chances are if the size difference is that large, cleaning up small files will be pointless. EDIT: If you're doing many to many for everything your processing time will be pretty damn big, talking about an acyclic undirected graph from everything to everything doing a longest common substring. The vis might be the least of your worries. :(

Answer 3

I think your current solution / thought process is too complicated - If you're only interested in how similar a file is to another file, it seems that visually representing file size wouldn't matter. You're only concerned with two things:

1. What files are similar to a file
2. How similar is each file to the current file?

1 - It seems from your examples that you think you're going to end up with a lot of similar files. Are you certain this is the case? If not, a more static layout may be easier to navigate than the ultra dynamic examples you provided. I can expand on this if you like and if you've got an answer to how many similar files you really think you'll be dealing with.

2 - If you're sold on circles you could use a simple pie chart representation of each file. Let's say your cut-off is 90% similarity, anything below that isn't close enough to matter. So, your pie chart only needs to represent 0%-10% (90-100). This is a small enough range to make the differences in the charts very noticeable. You could also tie color to this range. Start at something light and then get darker as they get closer to matching.

Answer 4

One potential solution would be to use a Self-Organising-Map (or SOM) - a map where the content of the file is used to determine it's location, and similar files end up near each other. This approach would be rigorous and reliable, but a lot of work.

(Example nabbed from here: http://jaanbi.blogspot.co.nz/2010/03/kohonens-self-organizing-maps-in.html)

The challenge with this approach is two fold.

First you need to autonomously (ie: without human intervention) generate a set of length n vectors to represent your set of files (this means automatically identifying the characteristics of the files that allow similarity and difference to be assessed).

Once you have this set of vectors, the other challenge is to map that n-dimensional space with a warped projection onto a two dimensional map.

I used to work on a system that did just this - you could feed it a corpus of many thousands of documents and it would generate a map showing the common themes and topics, with similar documents adjacent and dissimilar documents separated.

Answer 5

I recently solved a similar problem. I have thousands of mainly PDF files collected over years, and suspected that there were a lot of duplicate files.
My solution was not at all as sophisticated as the relationship maps mentioned in the other answers.
Using a command-line command I built a text file containing the file names and their sizes (and more ancillary data, as date), and sorted by size.
This single action joined plain duplicates, that I was able to detect because of the similar or equal file names.
I know that this is not quite an answer for this particular question, but I brought it up because it was surprisingly quick and effective.