Play is a triangle, reminding me of moving in some direction. Fast Forward and Fast Rewind are usually double triangles which somehow remind me of their intensity in their directions. Stop is like a block of concrete. I think of a dead-end seeing it. But each and every time I see Pause icon, I fail to understand it visually. Of course we all recognize Pause button immediatly everywhere we see it. But I think that's just because we're used to it. Is there any visual reasoning behind Pause icon?
Play symbol symbolizes moving forward. Fast Forward symbol is thus moving forward more. Rewind symbol is a backward play symbol, as it represents moving backward. The Pause symbol indicates no movement either forward nor backward. The stop symbol shows that there is no action of any sort.
Another good explanation from same link trying get into history of symbology
the pause button indicates the two rollers beside the read OR write magnet on a tape deck that push the tape up against the head. the single vertical bar with triangle indicates one roller retracted faster play in that direction... basically other than the "play" symbol, which simply means "go" the rest of the symbols are based on the state of the controlling rollers. Record was a red circle, indicating the red shelled "studio in use recording" light outside the door.
The vertical lines represent the sides of frames on a reel. Pause means you are stopped between two frames, play means you are moving through the frames left to right (hence the arrow), fast forward is moving through the frames at some multiple of 1x, and the scene skip button pushes you forward to some preset "hard" frame edge.
This one talks about how and where it originated
The symbols for play, stop, pause and record — the original media control symbols — were first introduced by Swedish Engineer Philip Olsson. Olsson was working in Japan while finishing his studies at the Royal Institute of Technology, having also earned a degree from a Swedish design school.
It is likely a symbol derived from the caesura sign used in musical notation to indicate a pause. But it's origin is not known for sure.
And here is an interesting comment from someone who claims to have seen the transition of this symbology.