The question of the usability of rounded corners versus square corners has already be answered here. My question revolves more about the consequence of having them intermingled. I've heard some people espouse the idea that once you go down a rounded corners path, your design should really stick to that approach rather then, for example, having a square box inside a rounded one. Is this simply aesthetic or is there some usability concern to back it up (e.g. cognitive switch from a softer less threatening to something more jarring)? I would be very interested in any research that attempts to answer this question.
Mixing rounded and square corners is usually considered poor design from an aesthetic point of view. From a usability perspective, the different kinds of corners would imply a difference between the two box types. If there really is a difference between the box types and the different corner treatments help communicated that, then that would tend to improve the usability.
I support the consistency issue but I think slightly different. Your browser viewport, windows and screen have no round corners. But content elements on web page could have it. Is it consistensy issue? Definitely no.
So my answer is to think of consistensy of logical elements' level. For example, all content blocks of same semantics are square cornered, but popup messages have rounded corners.
This is less an issue of rounded vs. square corners as it is an issue of consistency. Yes, you could do it, but your design would be inconsistent and look very much inconsistent - as if elements from competing designs were just thrown together.
Why don't we change the font for every alternating line between serif and sans-serif fonts? Why don't we mix sketchy and glossy items in a design? Consistency. Each option on their own works fine, but when you mix them together you are in effect highlighting the differences between them rather than having the various elements work together to form a consistent message.