Affordance has to exist, it is an implicit property of an object. Let me cite Norman (who is citing James J Gibson):
...the actions possible by a specific agent on a specific environment. To Gibson, affordances did not have to be perceivable or even knowable -- they simply existed
That's exactly the point, signifier without affordance is a mistake (a fixed door with a big PUSH HERE text) but affordance can exist without signifiers. One main point of UX design is to introduce signifiers to make affordance visible.
Norman used that word to identify perceived qualities of an object, this led to some confusions because now those terms slightly overlap but the central point is still the same: affordance is an implicit quality (visible or invisible) of an object to be used. When an object has not such perceived quality then you introduce signifiers to make it clear.
Let's imagine a secret door. According to Gibson it has the affordance to be—for example—pushed to be open even if you carefully tried to make it invisible and to remove that perception. Instead, according to Norman's definition, it lacks of affordance because its usage is not perceived.
Now imagine a normal door without handles. Because of materials, position or expectations (for example because it's at the end of an empty corridor) it has a perceived affordance. If it's not clear enough then you add a signifier, for example, a Push here plate.
Signifiers are not used just to make affordance visible, citing (again) Norman:
Consider a bookmark, a deliberately placed signifier of one's place in reading a book. But the physical nature of books also make them an accidental social signifier, for the placement of the bookmark tells the reader how much of the story remains.
To summarize in one line what I tried to explain in such dump of text:
The perceivable part of an affordance is a signifier, and if deliberately placed by a designer, it is a social signifier.