We use circular "radio" buttons to indicate there's only one choice available in the group.
Square checkboxes are used to indicate that a more flexible entry is allowed (none, one, or several). Individual choices are checked or unchecked.
If you're allowing none, one, or several options, use square checkboxes.
If you're only allowing one option, use a round option/radio button.
Allowing multiple choices for round checkboxes is just plain confusing.
Refer to the Microsoft guidelines for Radio Buttons for further reading.
With a radio button, users make a choice among a set of mutually exclusive, related options. Users can choose one and only one option. Radio buttons are so called because they function like the channel presets on radios.
With a check box, users make a decision between two clearly opposite choices. The check box label indicates the selected state, whereas the meaning of the cleared state must be the unambiguous opposite of the selected state. Consequently, check boxes should be used only to toggle an option on or off or to select or deselect an item.
In short, radio buttons offer a choice of a set of options (you have to listen to one radio station, you can't choose none or many). Checkboxes allow you a yes/no to each of a set of options.