My question is quite simple. Why can't we save images as a circle or a triangle? What is the reason behind it?
That's the convention because it's how images were made before computers were around, or even before camera's. Keeping in that standard pictures are generally the same as well.
To quote Nielsen's Usability Heuristics:
Match between system and the real world The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
In addition to this, PNG images can be saved as any shape you want really seeing as they support transparency.