...surveys, researches, scientific data?
Well, I will develop my answer only from a perceptual / formal point of view
The following parameters are relative (not absolute), everything depends on the context, and talking about flat buttons, no hyperrealism or skeuomorphism.
What determines that a shape is a button?
or what's the best that define a button?
Regularity
The more regular the shape, greater ease of perception.
Simplicity
In container and content
Oneness
Isolation
In the example images of the question, the first two points are well exposed: they are regular and simple shapes. As for the next two the result is more controversial.
The rectangular buttons leave a little margin by the characteristics of the shape, so it's more immediate to interpret it as a grid more than a group of buttons. This perception is accentuated by the content, it's a contact sheet.
As the statement of the question says that it's a group of buttons, the sense of uniqueness is a bit difficult to define, so isolation is the option to achieve.
Comparing both figures, the circle not only offers greater isolation space but also breaks the cartesian axes that establish the grid, strongly marked by the rectangles:
The final decision depends on the design guidelines, but from a formal point of view, in a grid of buttons, the ones with the best visual perception are the ones that provide the greatest isolation: between a vertical rectangle and a circle, the second one is the best option.
