User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
I want to represent an audio file in an image with a maximum size of 180×180 pixels.
I have been thinking about something similar to what Soundcloud does, although it will require some chopping and stretching to mash it into the square:
I wonder if any of you have something for this. I have been searching around for a bit, mainly audio visualization and audio thumbnailing, but I have not found anything useful.
Good question. In terms of UX there are a few different things you want in a thumbnail.
Thumbnails should be easily distinguished so you may wish to use some sort of visual hash (eg. here)
Similar items should look similar and different items should look different. Ideally you will want to encode a few things that are likely to remain similar across a sensible group of sounds. Here are some ideas as well as what they might help differentiate:
I imagine the naming (assuming these files have names) will be up to the user/consumer, this is the fallback if the user can't interpret the thumbnail so make sure it's clear and prominent enough just in case. (this problem appears with icons all the time, see UX Myth #13: Icons enhance usability)
Ideally the thumbnail should be memorable, though how to do this with sound I have no idea.
Finally you want all of this to be clear on your 180x180 thumbnail. Doing this well will doubtless be very challenging.
Perhaps something like this (though perhaps this is overkill or too cpu intensive):
Map the default hue to a mixture of some or all of the items in 2 above, this should give you similar color files for groups of files.
Frequency spectrogram in the background for saturation
Volume histogram can be represented by changing the hue of buckets (ie. squares in your 180x180 representation, 4 in my example, though you should use more)
Duration can be mapped logarithmically to a border surrounding the whole thing in the form of a bevel. (7 minute files will look almost exactly the same as 6 minute files, but completely different from 7 second files)
Here's an example of what it might look like:
Hope this helps
What about taking a waveform resolved to stretch only a short distance, say 200px, then wrap it around a ring so it fits inside the square thumbnail constraints. So... length of waveform graph would be... related to pi somewhat.
Like this, which I made in photoshop using the polar coordinates command.
A smooshed-up waveform doesn't seem very useful. For all practical purposes it's meaningless, and it'd be hard to tell two 180x180px waveforms apart at a glance. Instead you might consider a visual hash like Identicon. That would still be meaningless, but the thumbnails would be visually distinct. A clever algorithm could even make similar files (e.g. songs from the same album, or from the same artist) visually similar.
Digging around on Don Park's blog a bit turned up Frederic Brodbeck's Cinemetrics project, which may be useful. It's a visual hash intended for films that you might be able to adapt for use on audio.
I didn't understand whether by "chopping and stretching" you meant taking a segment of the audio data and stretching it so it fills the square, or rearranging it. The rearranging approach sounds like a good idea.
Thumbnails are meant as a preview for identifying the object without getting into the details.
Using the wave form in a thumbnail would not help identify the audio track.
What you could do is:
Use a vocal thumbnail (e.g. 10sec preview) played on mouse over
Use a visual representation of the contents e.g. use a database or pre-identified and tagged soundtracks or use audio recognition. Once you have identified the elements of the contents you could include small thumbnails that represent those elements in one bigger thumbnail.
E.g. a thumbnail containing a picture of the singer, a guitar or a picture of a train if the soundtrack is a sample of the noise a train makes.
The solution I settled for was to use Freesound's spectrogram generation code, then applying the Wadsworth Constant to the position for the thumbnailing. This gives you a snapshot of the audio clip with the center at 33% into the audio clip.
Example ↓
→ Live examples
The code for the process can be found at mediagoblin.media_types.audio.transcoders.py, line #99.
I will not mark any answer as correct, since none of them, not even mine is universally correct.
You might be knowing this already , this might work with the song name in center of the image or below.
Along the lines of @Forthright's answer, I think mapping the volume intensity to a gradient (colour/grayscale) and plotting along a Hilbert curve could be interesting.
This would be similar to http://corte.si/posts/code/hilbert/portrait/index.html and would provide a rectangular icon that gives an impression of the sound patterns that would be distinct for different files and scale to any length of audio and any icon dimensions.
There are some animal-sound libraries which use the first seconds of a recording as a thumbnail:
https://www.xeno-canto.org/explore?query=turdus+merula&view=3
https://www.tierstimmenarchiv.de/RefSys/Preview.php?CurLa=en
https://search.macaulaylibrary.org/catalog?mediaType=a
The last one also shows the full spectogram of the entire recording when you play the recording back on the website.
Spectograms contain a lot of information and a trained user can, for example, identify the species of bird in a recording by just looking at the spectogram (I'm speaking from experience here).
I find that spectograms are very easy an intuitive to understand once you actually try it, thus I hope more people start to experiment with them.
