I think people have learned about green checkmarks versus forbidden/deactivated symbols, so you might use that as an indicator:
The meaning of checkboxes to enable and disable things would probably work as well, too:
If you don't like two icons on one button, you could also integrate the meaning in the filter icon itself:
Do not solely rely on red/green color. The shape should also change, so that people with red-green blindness have a chance to notice the difference.