The choice of j and k is indeed inherited from the ADM-3A terminal by way of vi
(source). vi
is a popular enough editor among Google engineers that j, k, and a few other shortcuts inspired from it (for example / for search, y for yank) took root in Gmail and then YouTube (source on popularity of vim
at Google).
To understand where this design came from, especially the choice of J for "down" rather than K, a relevant document is the ASCII standard established in 1963. On page 5 of that standard, we can already see that the character corresponding to binary "0001010" (hex 0x0A) is the line-feed control character (marked "LF"), and the character corresponding to binary "1001010" (hex 0x4A) is J.
In many computers, even to this day, the effect of the Ctrl key is to input a character with code 0x40 less than the code of the character that would otherwise be typed, so that Ctrl+J would have resulted in a line-feed when sent to a printer (meaning that the paper would have rolled to the next line under the print head) and, by analogy, moving the cursor to the next line when sent to a display terminal.
In 1967, Ctrl+H (control character with hex code 0x08, since H is character with hex code 0x48) acquired the meaning of backspace ("BS") in the revised ASCII standard; it was previously "format effector" (FE0). We now understand backspace to mean the deletion of the previous character on screen, but back in the days of printing terminals, the role of backspace was to move the print head backwards one character to allow printing accents (so for example the sequence E backspace ’ would print something like É). It's a natural fit for ←.
The choice of the other keys doesn't map as clearly to the ASCII spec, for example Ctrl+K was "vertical tabulation" (VT), Ctrl+L was "form feed" (FF), but H and J were pretty much already assigned to backspace and linefeed, respectively, so that by the time the ADM-3A came out in 1976 it made sense to map them to ← and ↓. respectively. The choice of then using K and L for ↑ and →, which might have been novel to the ADM-3A, seems to result from a simple desire to lay out the cursor keys symmetrically.