How about something like this solution?
download bmml source – Wireframes created with Balsamiq Mockups
The idea is that your simple, single Follow button will change into a segmented button when it you are already in following mode. Clicking on the main part of the button will toggle if you follow at all or not, but in following mode, you can also click on the second segment of the button that will toggle the privacy mode. I do like the feature that the text changes from Following to Unfollow on hover, though you should consider that with increased use of touch devices, hover is not available everywhere.
I think the bottom arrow from Follow to Following publicly is something to think about. I think I would expect that to work if I just (by mistake?) clicked the main button area in public Following mode, so I can easily correct my mistake. But I would also expect that by default, following would be private. So, maybe you should only return to the public Following state if your just came from there, and not if it was longer ago. Say, for as long as you're on the page or something like that.