Information hierarchy applies to buttons too
Variable button sizes are an excellent way to reinforce action hierarchies. If every action was triggered by an identically sized button, your UI is likely to become a mess. Users would have a hard time finding the primary and secondary actions that answer their needs 80-90% of the time.
Create rules and follow them
You want to have a logical, planned system of tiered button styles. At the lowest level, you may have text in a different color. At the highest, you may have a brightly colored button with large text and large padding. Figure out where each style fits and use them consistently throughout the experience.
Here's an example from an app I'm working on showing primary and secondary buttons, as well as our bottom tier action for stepping back in the nav.
We also have a larger button for "critical path" actions, basic text links, and some actions in small modals that use the full column width.
Here's another example from a mobile workflow (generalized for the client's sake) where you can see a number of button types in one view. This is a dense UI with a lot of elements that, unfortunately, can't be reduced further. Identically sized buttons here would put unnecessary burden on the user to figure what the most important actions are at any given time.