I would like my users to toggle opt-in or opt-out the nested items in my checkbox tree individually but also toggle them all opt-in or all opt-out at once. In my example, I use radio buttons. But the problem is that for the all opt-in or out toggle there is no such thing as a mixed state with radio buttons like there is with checkboxes. On the other hand, checkboxes can check all opt-in AND out. Which would mean that all nested items could be both on and off and that's not what I would want.
I'm not sure if I understand the problem correctly, but I think you're oever complicating things. If you only need users to opt-in or out, this should be enough:
(you can use a toggle instead of checkbox if you prefer)
Now, if for some reason you need to have those checkboxes, an adequate title like Opt-in or Out and an explanation of what will happen should suffice to inform the user a checked checkbox means opt-in and unchecked opt-out.
Either way, there's at least one control that shouldn't be there and is redundant, unless there's some information you didn't provide and you need both actionable elements because they have different purposes. If that is the case, there's a chance you should rethink the UI to make those purposes different. Otherwise, we're back to square one and you need only one control.
The radio buttons don't really offer a commitment for the user and showing all the information at once might increase cognitive load. I recommend some progressive disclosure and validation.
Replacing the radio with switches "could" help users make it seem a more committed choice rather than the radio buttons while only showing the opt-in - opt-out option exactly when needed can move the focus of the user where it needs to be.
These are assumptions and it would be great to test it out with users but this is my recommendation.
What can help to find a solution is to think about the origin of the problem, leaving the design objects for later.
It's a table with selectable cells containing two options. Once the functionality and interactivity are resolved, it will be easier to find the type of element that best represents it.