As always, XKCD has an appropriate comic:

If you have a button on your website that is very rarely used, consider that the people who use that button might rely on it more than you think. For example, Word has an option to insert an ActiveX object. I doubt many people use it nowadays, but it's still in there because some people do.
Backwards compatibility is also important. You can open word documents dating back 15 years using the most recent version of office, but that also means it needs to support the features (and bugs) of 15 year old documents.
What I'd suggest is, instead of simply removing rarely used options, is to turn them into advanced options. Again, using Office, you can set often used options like font, size, formatting and emphasis using 1 group of 1 ribbon tab. if you want to do a strikethrough or a superscript, you just open an extra window using a small button on that ribbon group. You might have to educate the experienced users on where you can find it, but that's easily done using a help tooltip.