Take the 2-minute tour ×
User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It's 100% free, no registration required.

Problem

My file-system is a mess, over the years I've been accumulating way too many downloads I didn't care to uncompress and organize, code that was subject to endless copy+pastes and improvements and lots of images and documents that are basically the same file just with some minor changes/revisions.

I want to display the similarity of a file to any other file, as long as it is higher than the given threshold. I also want to do the same with folders.

Clarification: This is not about visualizing the differences between any 2 or more files, this is about how to represent the similarity of every file with all the other files (many to many).


Background

What I thought would be a good approach would be to have two kinds of perspectives: one that would show only files and another one that would show only files. Then I would display a circle (perhaps the circle dimension could be directly proportional to the file/folder size) for each node (file/folder) which I would connect to all the other similar nodes. The distance between the nodes would be inverse of the similarity between them, and the actions would be available through a context or side menu.

This idea has a couple of problems though:

  1. if there are many nodes, either the circles would have an impossible small area, or the distance between them would be minuscule or the scrolling area would be annoyingly huge

  2. if the node size was directly proportional to it's file size, then the above problem would be even more accentuated, or there would be some overlaps that would make the whole navigation unusable.

I was thinking of using something like this (but this one doesn't reflect the strength of the similarity) or like this (which has the problems I described above). Is there any design pattern that would fit here..?

I would like to do something like the Muffin Player:

muffin visionflow

But they have the advantage of being able to break their nodes into only 2 axis and they also have the depth dimension (although I'm not exactly sure about it's purpose).

share|improve this question
2  
What's your question? –  Rahul Oct 30 '12 at 13:25
1  
I suggest you update your question to make that more obvious. Right now it's a huge wall of text with no clear call to action for anyone to help you with. Focus on asking a question in the first paragraph, then supply supplemental information afterwards. –  Rahul Oct 30 '12 at 13:28
2  
I rewrote the question to focus on the problem instead of on your reasons for getting here. A good rule of thumb when writing questions is to get to the point as soon as possible and to only tell the reader what they need to know to answer the question. As such I removed a large part of your background, because it's not necessary to know how you went about using algorithms in order for a designer to help you. Also, try being as specific as possible: "I want to visualize differences between files", not "I want to visualize nodes in a graph". –  Rahul Oct 30 '12 at 14:01
1  
@JoshuaDrake: Yes, perhaps you're right. Lets assume I'm only diffing text files for now - I'm sure the rest will come along if this (major) problem is solved. –  Alix Axel Oct 31 '12 at 4:47
1  
The field of bioinformatics has developed some useful tools for doing the sort of large scale similarity relationships you are hoping to visualise. –  Stephen Oct 31 '12 at 21:15
show 7 more comments

5 Answers

As you mention a simple scheme alone would very likely run into information overload, with too many nodes overlapping one another. To my mind you need to take a combined approach, and likely start by allowing filtering of the search results, say to a given folder tree.

As for the combined approach I think you need to consider combining elements from the two following high-level areas.

Highlight the current file being compared.

Possibilities include:

  • Color - Make it red, or green or white or ...
  • Size - Make it the largest object
  • Shape - Make it a circle, square or triangle, although I like the idea of a target
  • Position - Make it central to the screen, or in the top left or right OR set it off in an explorer like view.

Differentiate all additional files

  • Color
    • Simple scheme: Red files are different, Yellow files share minimal content, Blue files are share a significant portion of content and Green files are identical
    • Shading: lighter as the files vary, the more they vary the lighter they are from the current shade. Possibly changing to another color along the spectrum as above.
  • Size - Smaller files share less in common with the currently selected file as objects that are farther away appear smaller.
  • Shape: Maybe star shapes are close matches, maybe pointy shapes are further away and rounded shapes are closer. This one could get vary subjective.
  • Location: files closer to the target more closely match it, could be combined with size above to give a familiar three dimensional feel. Like navigating nodes on a solar system chart in a console game.
share|improve this answer
add comment

If you're trying to visualize a file structure (a tree), and are trying to find ways to prune it, there are some things you can do.

The size of the blob needs to be associated with the size of the file.

The distance from the center could be associated with the distance from the root (C:)and on selection could be associated with the LCS (largest common sub-string) the longer the sub-string the closer the nodes should be.

The color could be associated with the file type association, media could be orange, executable blue etc.

Maybe you want to circle sets of related files (by folder) and show the size of a that. I know frequently those little files can add up and take up a ton of space, but the reason why you don't prune them is because you don't know where they're coming from.

if your differences in size are huge, allow the user to choose for example a log scale algorithm. EX: xkcd.com/482/

However chances are if the size difference is that large, cleaning up small files will be pointless. EDIT: If you're doing many to many for everything your processing time will be pretty damn big, talking about an acyclic undirected graph from everything to everything doing a longest common substring. The vis might be the least of your worries. :(

share|improve this answer
add comment

I think your current solution / thought process is too complicated - If you're only interested in how similar a file is to another file, it seems that visually representing file size wouldn't matter. You're only concerned with two things:

  1. What files are similar to a file
  2. How similar is each file to the current file?

1 - It seems from your examples that you think you're going to end up with a lot of similar files. Are you certain this is the case? If not, a more static layout may be easier to navigate than the ultra dynamic examples you provided. I can expand on this if you like and if you've got an answer to how many similar files you really think you'll be dealing with.

2 - If you're sold on circles you could use a simple pie chart representation of each file. Let's say your cut-off is 90% similarity, anything below that isn't close enough to matter. So, your pie chart only needs to represent 0%-10% (90-100). This is a small enough range to make the differences in the charts very noticeable. You could also tie color to this range. Start at something light and then get darker as they get closer to matching.

share|improve this answer
    
Thanks. I'm on the go right now but I would like to clarify something that was somewhat lost between the edits: my aim is to compare the universe of files with itself, not comparing each element on it's own with the entire set. –  Alix Axel Oct 30 '12 at 21:41
    
"Compare the universe of files with itself" - I have no idea what this means –  Rahul Oct 31 '12 at 2:57
1  
@Rahul: John suggestion would work if I was going to pick only one file and get a list of similarities to that same file. That's not what I'm looking for. If I have 10 files, I want to see how similar file 1 is to the other 9, how similar file 2 is to the other 9, and so on... Hence, compare the similarity of the universe (or set) of files with itself (the set). Makes sense? –  Alix Axel Oct 31 '12 at 4:43
    
@Alex: Ah, yes, the version I read made it seem as though you were only looking to compare a single file to many files. –  John S Nov 1 '12 at 17:26
add comment

One potential solution would be to use a Self-Organising-Map (or SOM) - a map where the content of the file is used to determine it's location, and similar files end up near each other. This approach would be rigorous and reliable, but a lot of work.

Sample SOM

(Example nabbed from here: http://jaanbi.blogspot.co.nz/2010/03/kohonens-self-organizing-maps-in.html)

The challenge with this approach is two fold.

First you need to autonomously (ie: without human intervention) generate a set of length n vectors to represent your set of files (this means automatically identifying the characteristics of the files that allow similarity and difference to be assessed).

Once you have this set of vectors, the other challenge is to map that n-dimensional space with a warped projection onto a two dimensional map.

I used to work on a system that did just this - you could feed it a corpus of many thousands of documents and it would generate a map showing the common themes and topics, with similar documents adjacent and dissimilar documents separated.

share|improve this answer
add comment

I recently solved a similar problem. I have thousands of mainly PDF files collected over years, and suspected that there were a lot of duplicate files.
My solution was not at all as sophisticated as the relationship maps mentioned in the other answers.
Using a command-line command I built a text file containing the file names and their sizes (and more ancillary data, as date), and sorted by size.
This single action joined plain duplicates, that I was able to detect because of the similar or equal file names.
I know that this is not quite an answer for this particular question, but I brought it up because it was surprisingly quick and effective.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.