Firstly, toggle buttons are somewhat problematic. Although they are a pretty common design pattern on modern touchscreen devices it isn't immediately obvious that one can activate them just by clicking. A user unfamiliar with this on-screen element but familiar with it's real life counterpart is going to try and drag it as we don't click such switches in the real world.
https://ux.stackexchange.com/a/43889/33233
https://ux.stackexchange.com/a/63886/33233
Also, if you look at sites like GMail or Amazon, who invest huge amount in UX you'll see checkboxes, not toggle switches.
Checkboxes mean on / off (yes, or no; or if you're geeky, 1 or 0) and have been used on the web literally billions of times to represent yes or no, on or off type choices for users.
You do also encounter a problem if you were to use the same element for the parent and the child (particularly a checkbox) as you imitate another very common web design pattern which is the all or none parent checkbox pattern. This goes beyond the aesthetic issue you raise in your question. There are 1000s of examples of this pattern throughout the web.
What I would recommend is the following design:

download bmml source – Wireframes created with Balsamiq Mockups
Where the on and off selections are clearly clickable and where the checkboxes themselves would remain set as the user left them. The current state would be clearly defined by which button was selected
Also, the entire panel would be either greyed out, collapsed or both while in the off state. Both of these actions and the switch in interface state depending on choice would help to reinforce the action.
PS ... I am assuming you want the checkbox set state to be retained regardless of the parent state, if not and the parent is a 'toggle all' function then use checkboxes throughout as this is a very well established, common and usable solution.