Generally, buttons shown together are also visually similar.
Are there situations where it might make sense to show together buttons of different shapes? E.g. a rounded rectangle button together with a circular button?
As a crude example to analyze, here is a sketch.
I suggest you to do like the most websites. Same shape, side by side, different colours. Highlight and put first the main used button. See good examples below.
I know that you ask just about the format and my answer is a bit more complex but this definitely could help you.
Make buttons intuitive changing their positions
More buttons, more option, more decisions. Easy to take no action or the wrong action.
We should influence the users to take the right action giving the right priority for each link or button. We can use font-weight, font-size and colours for it. Of course, the biggest button should be the most important.
The position is absolutely important to help user preventing the reverse page scanning. Reverse page scanning is not good and breaks the natural flow, generating unnecessary scrolls and possibly will result in doubts. And again, the user with doubts or too many options tend to take no action or the wrong action.
See some examples:
PS: It's a mobile example, but works quite similar on desktop. The most important thing is the concept.
I found these images a few months ago in the UX Movement post but unfortunately, I did not remember the name.
Consider the metaphor you're using and its roots. Buttons. What are buttons? They're an element you can "press" or "click" to take some action. Where does this metaphor come from? From physical machines.
Do physical machines have buttons of different size or shape near one another? Yes! Look down at the keyboard in front of you. Assuming you aren't using a grid-layout Planck or Preonic or other otholinear, you'll notice keys with 1u (one key-unit), 2u, 1.5u, 2.25u, 6u, and maybe other lengths all on the same board. If you look at the dash in your car or the HMI in an industrial setting, those typically also have buttons of different sizes and shapes (not to mention color, height, texture, labels, etc.).
So, yes, I think that sometimes, perhaps often, it can make sense to have buttons of different shapes (including buttons that are variations on a shape or from the same shape family), especially if you have many buttons present or wish to differentiate the buttons to a greater extent.
There are limited shapes we can use for buttons - rectangle, rounded rectangle, circle.
If we use too many shapes for buttons, it wouldn look more like icons rather than buttons.
It is necessary to keep consistent shape for action buttons as people generally scan interface and consistency helps to fix user attention towards action. Again, if some buttons have pre defined meaning in user's mental model like Arrow shapes for navigation, rounded button with + for adding something, square button with + to expand, then it makes sense to use these shapes.
However, buttons with text on it as action should be consistent in shape. Their state, affordance, availability can be depicted through font/color effects.
If buttons are semantically/functional related, then they should be in close proximity regardless of their appearance.
An example would be a window/screen that has a 'cancel' button and a 'submit' button. A common practice is to place them at opposite sides of the window/screen, but that's not ideal because people have very limited central vision.
Even though each button has a distinct function, they have a common effect that's much more palpable: they both close the current window/screen. They should be beside each other in the center, although one could be made to look more salient.
Think of the buttons in this example as siblings. They should go together because of their underlying commonality despite maybe not having identical appearances.
