The simple answer to your question is that first and foremost, they are distractions. This isn't about their utility or importance, but just how they affect the user, which is that by definition, notifications - whether pushed to your phone or flashing on your toolbar - are intended to interrupt them.
The goal of these interruptions be their nature is to inform you about something of perceived importance, at the very least, and/or get you to take action on something that needs it. Without this criteria a notification is just a nuisance. But if we think through that, what happens?
If the notification is purely informative, you still are reminded of it, and it ever so slightly takes up your cognitive load as your mind registers the notification, processes it and dismisses it. This mental load adds up with each new ding or flash and detracts from your productivity and your thinking, which is why any guru or blogger discussing the subject of productivity at work, or meditation, or just getting in "the zone", will tell you to remove all distractions. Even leaving the notification alone, i.e. watching that number pile up with unread emails, is a constant source of agitation because users are so used to getting rid of that, and it literally becomes an addiction.
Needing to take action on a notification is even more of a cognitive load, even in its most basic form of confirming an alert, say. More involved alerts that require you to switch between tasks or apps force you to spend time multitasking, which has been thoroughly proven to waste more time than it saves. Read "The Myth of Multitasking"(.pdf) as an example.
That is the drawback of notifications, and why turning them off has become so common. Of course, what is very important to note here is context. Since you mentioned the iPhone a few times I'll use their push notifications as examples but the principles can be applied anywhere:
Importance: a lot of apps use notifications as nothing more than marketing vehicles to remind users to use those apps. Trivial in their own right but a constant distraction if not turned off.
Frequency: I would surmise Facebook gets a lot of loathing because that is a type of app that has the potential to send you a flood of alerts, by the minute: for chats, for new posts and tagged photos, etc etc. It doesn't matter how important the content is - you wouldn't want to be reminded of it every two minutes.
Priority: If you have one app on your phone that sends you an important notification, you might decide to keep it because it is indeed important enough to you that it outweighs the distraction. If you end up getting 5 more apps, each with their own alerts, you start reevaluating that mindset, and start prioritizing what is the most important among all apps. That original app's alerts in this new context, might not be as relevant as it was.
So in essence, every user will react to the same notification differently, and to reiterate, I don't want to imply that all notifications are trivial or unnecessary, but the takeaway is that they are intended distractions. Whether that interruption is worth the value of whatever info/task that alert is providing, is up to the user to first realize then decide.
