The files which are deleted are in fact recycled. The material from which they are made -- storage bits -- is regrouped and reused for other files, just like recycled plastic can be used to make something new, like a synthetic fiber sweater.
The metaphor is much older than Microsoft Windows.
"Garbage collection" was introduced in the late 1950's in the Lisp programming language.
But in fact garbage collection performs recycling (the term typically used is "storage reclamation"). Digital objects which have become "garbage" are turned into storage for new objects.
Memory and disk storage are a "serially reusable" resource: when a program no longer needs it, another one can use it.
Recycling of computer storage is in fact a more perfect form of recycling than, say, plastic or paper which are usually only "downcycling": recovering an inferior form of raw material compared to the original.
The recycle bin doesn't actually recycle anything, just like a real recycle bin where items sit until the real disposal takes place which initiates the recycling process. Just like a real recycling bin, the Windows one lets us change our minds: the same way we can take back a plastic bottle from our recycle bin, we can restore a deleted file. This restore action is not understood as recycling: in computing it is called "recovery" and in physical disposal it is usually called "reuse", rather than "recycling". (Though, by metonymy, sometimes the reuse of materials is referred to recycling; something made of re-used objects which were never actually thrown away is sometimes said to be "made of recycled goods". However, these goods are not reused for their original purpose. If we use a plastic beverage bottle for carrying drinking water, that is not called recycling.)
When we "empty the recycle bin", that is analogous to someone taking away our recycled material to a recycling center. At that point, the opportunity to recover a file is lost, just like the opportunity to reuse a plastic bottle.
Deleting a file without going to the recycle bin (for instance, using Shift-Delete in Windows Exporer to delete permanently) is analogous to taking an item directly to the recycling depot, bypassing the local recycle bin. Once an item is handed over to the depot, it cannot be recovered.
The recycling analogy holds very well because computer storage is a real, physical resource; the recycling is just as real as that of plastic. Storage bits are real three-dimensional objects. These bits have a location in space, mass, volume and so on. Why we have to recycle them is that there are only so many available in a device, which has to do with physical constraints. There being only 32Gb on a flash chip is no different from, say, there only being so many liters of water on a planet.
So, Microsoft has considerable technical justification in calling that system feature "Recycle Bin". Yet, it has been called "Trash" or similar in other systems. Perhaps, part of the motivation is simply to be different and to tip a hat, so to speak, to the recycling movement. The recycle bin says "I am of this age; I think about the environment, and encourage people who look at me to do the same".