Why are circular buttons in software/websites so rarely used? Is it purely convention or are there usability reasons behind this?
Circular buttons can work well when an icon is all that's needed inside. A good example of this is Path. Their single icon buttons are circular and work well. However, to fit a word in the button, like 'Register', the button would have to be as tall as it is wide, taking up a lot of valuable space and creating a huge button. If using completely circular buttons throughout an entire interface, all buttons would have to be as large as the longest word(s) within a single button. IE, a button for the word 'Add' would have to be the same size as the button for 'Register', for continuity in sizing throughout the interface. This would be a huge waste of space.
Lots of apps like Path and sites like Svbtle get around this by using circular buttons for icons, then keeping the same height button and elongating it horizontally to create a rectangle with rounded corners for buttons with text.
I guess the disadvantage in terms of usability is that a circle or a rectangle with circular rounded corners has a smaller click area than a rectangle with smaller or no rounded corners. So, there's a slightly increased chance of users missing the button completely. However, this is more of a style choice I think.
The answer is largely historical. In early HTML, there were no native circles or even squares with rounded corners, and the only way that you could have them was to use images. And at a time when speed mattered a lot (think 2000+ times slower than a connection today), most websites avoided images as much as possible.
So, if you wanted to make a button, you used a rectangle. I suppose it stuck as people became used to buttons being rectangles. There are many situations where a rectangular button is a better choice than a square one, but not so many to justify the rarity of a circular button.
When you look at physical interfaces, however, you will see that circular buttons are very popular. They are in general easier to make, cheaper, and arguably more aesthetically pleasing. I think if you watch interfaces over the next few years, you will see a marked increase in the use of round or non-rectangular buttons.
Considder how you're going to place multiple round icons:
In the first solution, you're basically still using a rectangular grid, and you're effectively wasting the space around the icons. However, with circular icons you could go for a hexagonal grid like in the bottom solution. That would allow placing the icons closer together than in a rectangular grid, but I doubt it will score well on scanability.