While it depends on your demographics, I would say yes, you should translate it. It also depends on whether you are translating everything else. Having a translated control in the middle of other untranslated controls (or content) is somewhat confusing and definitely looks bad, as if the designer / developer did not care.
About your specific question: we conducted a test for an application in Spanish. We are in Argentina and we have a wide influence / use of Anglicisms in the words we use every day. So, just as you say, we assumed that
on / off would be easily understandable. And so it was ... for the age segment of 14 to 50. The older the age, the bigger the friction.
But then it got worse: when considering other Spanish-speaking countries, friction with
ON / OFF was really important in almost all age segments. This was seen especially in Spain, which is usually a more careful country with the influence of foreign languages.
Anyways, in Spanish,
ON / OFF is translated as
ENCENDIDO / APAGADO. And our particular case, we did not have space for so much text, so we decided to use ON / OFF and help us with a
switch where both states are easily distinguishable. A good example would be Material switches
Or something where the on/off wording is clearly supported by its affordance, like this, where they use color an iconography on top of the
Bottom line: if it is within your possibilities and you do not find space limitations, I would recommend that you translate the labels