The accessibility guidelines from the W3C offer neat and usable guidelines for creating text that is readable for everyone. There are a number of very useful tools for checking contrast levels, so it's pretty easy to at least get the basics up to AA level compliancy.
However, it doesn't become clear from the texts what to do with buttons, icons and other visual control. These are different from text, are used differently, and this leads me to think it's possible to be more lenient about contrast here.
Of course, if you want to be on the safe side you can aim for a 4.5 contrast ratio here as well. However, usually we also try to make things pretty and not simply lowest common denominator accessible. Also, having high visual contrast on everything also impedes usability, especially in user interfaces that are necessarily cluttered.
I usually look at the big guys for guidance. Apple, Facebook, Microsoft etc. usually make sure they're accessible, and for instance have text and links at a proper contrast. However these two shots are from iOS7:
ios7 calculator screenshot
ios7 settings screenshot
The orange to white contrast in the buttons does not comply to AA level, not even for large text. I would've thought that would make it hard to distinguish what sign is on the button for many people. The same goes for the green in the slider. Wouldn't that make it hard for many people to use? The increase contrast option in iOS settings does not affect these at all. They offer a 'labels' option for the on/off slider, but that hardly has any contrast at all. The only way to "solve" this is to switch to inverted colors.
So, WCAG being fuzzy on this, how do I go about setting a benchmark for my design team to create accessible buttons and icons?