Ideally, we'd always be able to give the user an estimate of the amount of time remaining. Visually, this is usually done through the infamous progress bar. However, certain activities such as waiting for a stream to buffer are difficult to estimate completion for. Most, if not all, progress bars have some sort of "indefinite" state available to programmers which give a candy-bar effect alerting the user, "I have no idea how long this will take." (I'm thinking specifically of Java Swing and Twitter Bootstrap, but I'm aware there are others.) However, seeing a progress bar that's not displaying progress is infuriating. "Why doesn't it just show me how much time is left?"
The spinner relieves some of this stress with a visual form that leads itself to cyclical animation. Rather than a bar that's expected to fill, the user sees a wheel and expects it to spin. (Reversing this relationship, you could also fill a circular form to display concrete progress.)
As another plus, spinners are easily recognizeable (thanks iPhone) and can be used at lower resolutions without losing their impact, allowing for more flexible UI design. For instance, some designs use action buttons that, when clicked, show a spinner right in the button to denote activity.
An interesting note is that, with good design, many items that typically use spinners could be modified to use progress bars instead. This would be more informative to the user. For instance, when loading a file over a network, the HTTP header often contains within it the size of the file. This could be used to estimate progress. My guess is that people tend towards spinners for their flexibility and their "modern" feel.
\|/-
characters) as well. I would imagine "replace last character" is a much easier animation than any other. This probably had a lot to do with the evolution of the spinner.