Known as contour bias (see this page from the book Universal Principles of Design), rounded corners make objects appear less harsh and more friendly. The book cites the seminal work on contour bias as being this article Humans Prefer Curved Visual Objects by Moshe Bar and Maital Net
However, note that rounded corners are not necessarily the right answer - the book also says:
Objects with pointed features, elicited stronger activations in the
region of the brain related to associative processing, meaning that
although angular objects were less liked they elicited a deeper level
of processing than did the contoured objects - they were in effect,
more interesting and more thought-provoking to look at
So perhaps it is a question of whether you wish content to appear more friendly or more noticeable. As the book continues:
Angular objects are more effective at attracting attention and
engaging thought; contoured objects are more effective at making a
positive emotional and aesthetic impression.
I was thinking about what we find in nature - and subconscious interpretation.
We are preconditioned over time to what we find in the natural world. Curves are more likely found in nature, while square corners are pretty unlikely, and often where they might occur (eg due to fractures, breaks or other forces) they get worn down over time to make a natural curve again. Square corners simply do not have a place in natural world.
For this reason, when we try to create a natural looking page we create rounded corners rather than square corners. This helps the border or the frame to sit more comfortably in the content. However, if we use square corners, those sharper points become more noticeable (the subconscious is hardwired to look for danger and threats).
Square corners start to distance themselves from the content. The result being that square frames are interpreted more as a separate window - providing a view through into the content inside. Little surprise then, that art is hung in a square frame - the frame doesn't get seen as part of the picture - which allows you to see the whole picture unhindered. It's about separation of figure and ground.
For this reason, I'd expect the message inside a square button to be more clearly perceived if inside a square cornered frame than a rounded one. Indeed, a serious message or warning does seem to carry more importance and command more respect in a square shape than a friendly rounded one. And no surprise that even more pointy triangles are used for warning signs.
So - it seems to me that both square and rounded have their place depending on context and how the message is intended to be perceived - whether the message is serious and needs to be presented unhindered for maximum impact, or a natural and organic, friendly and inviting environment is more appropriate.