I don't see any common use case to show both the inches and millimeters at the same time, while showing them at the same time may cause confusion.
Visuals such as colour aside, I would keep the switch and the question, but only show the "Size in inches" if the switch shows inches, and similarly for millimeters. As an aside, I would use "inches" not "IN" on the switch, as IN isn't a standard short form for inches, and the standard short form is simply
" which is unlikely to be clear.
The buttons should then use something more clear like
Use inches /
Use millimeters depending on which switch has been set. This gives a mix of maximum clarity while still maintaining expectations.
There are a few ways of handing the selection design wise, and I've only given one that is as close to what you have now, while still being good UX wise. In the end, you really need to test out whatever your choice is with users to know what is right for your situation.