I'd stop just short of "don't use them."
I'd suggest toggle buttons are acceptable in the case where there is a clear on and off state. This can occur, for instance, when you have a line of grayed buttons that become colored when you click them.
This is the reason Play/Pause works in many cases. The play button is not so much a toggle between two states as an enable/disable. It turns bright green when on (I'm playing!), it turns gray and shows a || when off (I'm not playing!).
The other buttons do the same, so the consistency adds to the affordance. It's probably also based on our historical perceptions of a tape deck or VCR, which had buttons you pressed in (i.e. on state) or depressed (i.e. off state) and I think this is a great mental model to keep in mind when considering a toggle.
But in any case where they are not simply "activated", where you couldn't replace the toggle with an uglier checkbox, I'd default to "don't use them," for all the reasons already mentioned.
ADDITION
I just noticed today that Evernote has a pretty effective toggle. They took the approach of a mouseover, which actually worked pretty well. Note that this is another case of an on/off style control.
ADDITIONAL ADDITION
I found this interesting switch today and was reminded of this UX Q&A. Note how the state is immediately obvious, and the fact that it's going to switch when you touch it is also made obvious by the textured "handle" (i.e. affordance)--I just know that when I grab that raised knob, it will switch to the opposite side and say the opposite of what it says now; it's just like an airplane lavatory.
So they can be done well.