If I understood correctly the premises of the question, what the user should decide is:

 1. If the user wants to consider (filter by) one of the following options
 2. If the answer to 1 is Yes, then choose one of the options

The possible outcomes are:

 - Option A
 - Option B
 - Not applicable

**Original proposal**: [[demo]](https://jsfiddle.net/4nh2795s/show/)

Considering you are using [toggle buttons](https://material.io/guidelines/components/buttons.html#buttons-toggle-buttons), for the third outcome you could give the possibility to unselect both options (adding an extra note to clarify this possibility or an *"(optional)"* next to the label). Either one or non buttons can be selected. Clicking on an already selected button deselects it.

<img src="https://i.sstatic.net/6W3J3.png" width="280" />

**Second proposal**:
Let the user decide 1 and then 2. The checkbox enables/disables the buttons. In this case the toggle buttons can act as radio buttons where one needs to be selected.

<img src="https://i.sstatic.net/rQmG9.png" width="280" /> 

(Thanks @theonlygusti for the feedback on the checkbox wording)