My interpretation is that people are getting used to the new world of computer interfaces and they don't need metaphors to hold their hand anymore. Something doesn't need to look exactly like a button to know you can press it. We need far less information to recognize the buttons, so we can now get rid of some of the noise.
It a very common pattern when new technology emerges. Here's a very old example:
At first, the symbols are direct (if cartoony) representations. Then they slowly lose their extrinsic meaning and become defined more by our conventions of using them (our shared language) and the references to outside concepts disappear.
This is what's happening in flat design. We aren't losing the affordances per se (if the design is good), we're just using a common language of more subtle cues to indicate the function of an object.
Note that true affordance (in Norman's original sense) is very rarely found in interface objects. A door may prohibit us from pulling it by not providing a handle (and only a push plate), but interface elements can rarely be so forceful. They communicate using simpler cues. For a long time these were based on more familiar technologies, but as computer interfaces become more common and tangible interfaces less so, these references are becoming less helpful.
The language of flat design relies on cues like high contrast boxes, rounded corners, use of color and content-based navigation. I wouldn't think of it as a reduction in affordance, but as references to old technology becoming less useful as metaphors. I would bet that any child born today (in a first world country) pushes a digital button before she pushes a tangible, old fashioned one.
New technology is always phrased in terms of the old. When the telephone was first invented, it was pitched as a news service that you could call to have your newspaper read out to you. At some point it breaks free of these constraints and begins to be seen on its own terms. That's what's happening with interfaces.
So my answer would be: flat design is a response to less need for affordance and a reduction in the efficiency of interface metaphors.