In a word: dysiconia. That's my own coinage, by analogy to dyslexia, and means that I have a really, really, really hard time trying to figure out what the stupid little pictures in a typical GUI are supposed to mean. I know they're supposed to be intuitive, but to me they're not. I think this might be true of more people that you'd suspect, or why else do GUIs often add text below their icons?
Of course with extended use I would memorize a few common icons (though I still can't understand why they're supposed to be 'intuitive'), but confront me with a new one and once again I'm clueless. Worse, there's no good way to look up a picture. With an unfamiliar command, I can simply run "man strange_command", look in the index of the program manual, or use a Google search.
Humans are naturally verbal creatures - after all, language is what's supposed to distinguish us from the rest of the animals, isn't it? For those of us who use alphabetic languages, a good reading vocabulary in our native language would be upwards of 50,000 words. (Those with multiple languages might easily add 10K or more per language, depending on fluency.) We can either look up unfamiliar words in a dictionary, or deduce their meaning from context. Adding the few thousand (not hundreds of thousands) needed to use a CLI to this base is easily done.
All this is to say that for most literate* people, using a terminal/CLI is just an easy extension to skills they already have. Once one becomes familiar with with it, it is generally superior to graphical methods for any task that isn't inherently graphical (such as drawing).
*I suspect the popularity of GUIs goes hand-in-hand with the increase in functional illiteracy. Likewise voice input & output.