Wikipedia seems to be popular with the information so far... :)
So here's info from the history section of the page on caps lock
The Caps Lock key is a modified version of the Shift lock key that
occupies the same position on the keyboards of mechanical typewriters.
An early innovation in mechanical typewriters was the introduction of
a second character on each typebar, thereby doubling the number of
characters that could be typed, using the same number of keys. The
second character was positioned above the first on the face of each
typebar, and the shift key caused the entire type apparatus to move,
physically shifting the positioning of the typebars relative to the
ink ribbon. Just as in modern computer keyboards, the shifted position
was used to produce capitals and secondary characters.
Because the shift key mechanism on a mechanical typewriter requires
more force to operate and is usually operated by the little finger on
the left hand, it was difficult to hold the shift down for more than
two or three consecutive strokes, therefore the introduction of the
Shift lock key also helped out people with disabilities who could not
hold down more than one key at a time. The Shift lock key was
introduced so the shift operation could be maintained indefinitely
without continuous effort. It mechanically locked the typebars in the
shifted position, causing the upper character to be typed upon
pressing any key.
As described above, the Caps Lock behaviour in most QWERTY keyboard
layouts differs from the Shift lock behaviour in that it capitalizes
letters but does not affect other keys, such as numbers or
punctuation. Some early computer keyboards, such as the Commodore 64,
had a Shift lock but no Caps Lock; others, such as the BBC Micro, had
both, only one of which could be enabled at a time.
A version of Caps Lock that behaves like a traditional Shift lock does
exist on certain layouts such as the French AZERTY. Some operating
systems and window managers allow Caps Lock to be used for a similar
function. This behavior of the Caps Lock survives, however, in German
and Austrian QWERTZ keyboards.
So I tend to agree with one of the comments on the question, that the down arrow is indicative of the holding down of the shift key, as has historically been the method to obtain capitals.