What is the best way to represent audio visually?

Question

I've only taken some basic signal analysis courses, so I might be missing some things.

Purely theoretical question:

What methods exist for representing audio?

What methods could be made for representing audio, more specifically musical audio?

So far, I'm aware of:

• Viewing the waveform (Soundcloud does this), mostly useless except for seeing "loudness"

• Spectral analysis (Example), good for seeing frequency and "loudness"

Essentially I'm wondering if there is a way one could "see" the notes, beats, and so on of a song, visually.

Right off the top of my head I can think of displaying 3 differently colored waves over time representing treble, mid, bass in a soundcloud-like container with the section playing (or moused-over) being magnified, with the surrounding waveforms being compressed into the corners (like a wide-angle lens effect).

EDIT: I don't know where this could be used, this was just born out of my frustration with current audio visualizing technology.

I imagine having a 3d graph of a spectral analysis over time would be the "best" solution since you see everything but it might not be the most elegant and it might not be portable to places like soundcloud.

Even current spectrum analysis is hard to decipher (Too low level for images):

I'm essentially wondering what might work for casual users, and for people wondering ahead of time how the song will play out.

THIS HAS BEEN CROSS-POSTED TO DSP: linksauce

Answer 1

Bit of a wild-card (two, actually), but have you seen this?

Music with color and fire

At the risk of sounding even more idiotic, I think Disney could be a good source of inspiration (have you watched 'Dumbo' or 'Fantasia' recently?).

Answer 2

To be able to visually identify elements in the song, you will need to perform some sort of pattern matching between time frames (filtered windows in technical lingo) and a larger number of reference samples.

This process is indeed better asked in http://dsp.stackexchange.com, however, assume this can be dones, you could use a predefined visual symbol for each reference samples, then show a time line with the identified symbols (and this returns the answer to the domain of UX).

Answer 3

I guess there are two problems you're asking; how to make a sound file look like sound before it's playing, and how to display audio while it's playing.

I can't help much in the first case (although the waveform a you mention is relatively intuitive as an amateur), the convention I'm aware of for the second case is a VU meter (either a digital representation of an old-school needle moving rotationally, or a simple field or a few that fills/lights up as the audio plays).

Those are often used here in Australia on newscasts that have phone audio playing to denote "audio only".