(I encountered this on a project last year.)
While I really like the idea user-generated badges (they are much more personal), you also lose something in terms of authority or "specialness" when they are not official and sanctioned by the system. Also, how do you maintain the "quality" of nicely designed badges when anyone can create them? One compromise is to suggest/promote ideas for new badges, that then get "approved" and rolled out (complete w/ some polished artwork created by a paid designer). This keeps badges from being too focused on a particular demographic (
> foursquare) but allows badges to still be set aside as something special and something of high quality.
The safest route is not to start with user-generated badges, then slowly open up that gate. Also, instead of an either/or, make that feature (creating badges) one that is unlocked as a privilege. So, only active (responsible) users can do this. Or, make it cost something--especially if the badges carry some weight to them. For example, I could create a badge for blowing your nose, or a badge for completing an Iron Man, they are not at all the same. Try making these badges cost the user something to create, then they're likely to be more meaningful.
On a related note, I believe the iPhone app "EpicWin" lets you create your own goals/rewards. Consequently, I've heard of some people creating 3,000 points challenges for stupid things, to show how absurd that is. Not an issue as this app is personal, not social. But, good to look at that to see what people do. I think in a social setting, you need some arbitrator (the group or system/company) to enforce some parity among different badges.