It's linguistically consistent because of specificity.
"I want to buy a car" includes all cars.
Only when you specify extra details they get taken in to consideration. "I want to buy a red car" or "I do not want a blue car" both add color as an extra chunk of information, and that's when it becomes useful for filtering. Before that moment the color data isn't relevant.
Nobody goes and initially says "I want a car that's red or yellow or green or blue or purple or pink or white or grey or black". That's unnecessary information, and just as unwieldy as having all boxes ticked by default and having to untick N-1 boxes if you just want one color.
That's not to say the most-used filter system is completely parallel to how we would verbally communicate. Generally we'd go up to a car sales man and ask for a car, and they might ask if the color is important. Or they'd show us a blue one and then we'd tell that we prefer red.
To mimic that interaction we would have a separate tickbox for 'filter by color'. If unticked show all cars, if ticked show a toggle for include/exclude and then some colors. But we skip that initial tickbox because having nothing selected implies that it's not important either way and we can just run a script and save the user a click.
I should note that it's okay to use that first tickbox to clean up the interface a bit (by using accordions) or have a dropdown like google images that defaults to 'any color'. But you really really want to avoid having 10 different tickboxes selected by default because, like you said, people will have to untick all of them.
In short; no user input means 'no filtering', not 'filtering but include every option'. Though the shown endresult is the same, the logical steps towards it are different.