If I understood correctly the premises of the question, what the user should decide is:
- If the user wants to consider (filter by) one of the following options
- If the answer to 1 is Yes, then choose one of the options
The possible outcomes are:
- Option A
- Option B
- Not applicable
Original proposal: [demo]
Considering you are using toggle buttons, for the third outcome you could give the possibility to unselect both options (adding an extra note to clarify this possibility or an "(optional)" next to the label). Either one or non buttons can be selected. Clicking on an already selected button deselects it.
Second proposal: MakeLet the user decide 1 and then 2. The checkbox enables/disables the buttons. In this case the toggle buttons can act as radio buttons where one needs to be selected.
(Thanks @theonlygusti for the feedback on the checkbox wording)