It is confusing and there apparently isn’t any standardization or general human tendency. For example, MS Windows UX Interaction Guidelines specifies four basic kinds of toggling progressive disclosure control. Three out of four show the state-when-activated, while one shows the current state. I suspect this is a controversy that needs some innovation and research to resolve.
Toggle buttons are an established control in the Gnome, where they appear to be functional synonyms for option (radio) buttons. They are also described in the Apple Human Interface Guidelines, along with very similar "segmented controls," which are often used to set a view or presentation. IMO, toggle buttons should specifically be used for what you’re talking about: starting and stopping processes. This might be the way to resolve the controversy.
For use to start and stop a process, a toggle button should be labeled unambiguously with the action that yields its affirmative (running) state irrespective its current state. For example, for an email client, it should be labeled “Connect,” not “Online.” The current state is indicated by the toggle button’s graphic appearance, not its labels. When Off, the toggle button should look “raised,” so it appears like a command button. When On, the toggle button should “stick” in a sunken state, like a state indicator (e.g., a read-only text box).
For minimal ambiguity, you want two mutually-exclusive toggle buttons (e.g., one to Connect and one to Disconnect), consistent with the Gnome guidelines. However, this consumes double the real estate, and I suspect you can do okay with just one toggle button for processes like Connect that have simple True and False states (analogous to using a check box rather than two option buttons).
The appearance and behavior of a toggle button is consistent with physical toggle button switches (like the Play buttons on older physical tape recorders you allude to). It’s also consistent with option buttons, check boxes, and state menu items, which all show their affirmative states through standard graphics. When the control looks like a command button (raised appearance), it’s labeled like a command button, the label indicating the action committed like any command button (Connect). When it looks like a state indicator (sunken), its labeled suggest its current state (connected).
An alternative is swapping labels to indicate action/state like you describe. This works for ordinary command buttons if you use text labels that unambiguously indicate the action committed on activation. Don’t use an icon and don’t use state labels like “Online” and “Offline” because command buttons indicate their actions, not states. Use labels like “Connect” and “Disconnect.” The disadvantage of this approach is that you may get longer wordier labels in some cases and the user needs to do a mental transformation to read the affirmative state (“It says I can disconnect, therefore I must be online now”).
Swapping labels in a state indicator control like text box should have unambiguous labels of their state, not the action committed (“Online”/“Offline”). The problem with this approach is that state indicators do not traditionally execute commands, so users may not think to click on them to start/stop a process.
A check box control can be used for in some cases, but that seems more appropriate for changing an attribute value rather than starting and stopping a process.