Visually of course, they serve to communicate information. If just purely visual elements, then certainly you have some freedom to use different sizes.
However if they are buttons and clickable, they should all be the same size. Consistency in UI is part of overall design and UX. Consistency in size and style teaches the user what is a button, and what is not a button. Consistency allows users to consistently click icon buttons. Buttons that are smaller slow the user down and require then to more carefully position the cursor to click. And if part of a touch interface, small buttons can be very difficult to click.
You can see an example of how Material Design specifies general button sizes rather explicitly at https://material.io/components/buttons#specs and floating action buttons at https://material.io/components/buttons-floating-action-button#specs
For some reading about the importance of consistency in UI, you might check out https://uxdesign.cc/design-principle-consistency-6b0cf7e7339f