Combining two separate toggle buttons (follow/unfollow + another for whether that "follow" is public/private) seems like an elegant solution to the spacing and aesthetic problems of having two buttons.
I've never seen this before, so I'm either innovating or pushing my users off a cliff.
Will the small learning curve that's not perfectly intuitive be worth the benefits?
Does this seem easy enough for users to understand?
The following is my solution. Using the regular follow and following button, add an additional icon (doesn't have to be an eye) to distinguish whether the 'follow' is private or public.
A button-like object should not have three states.
Updated
Another solution I thought of was a slider-like function:
When the user has it selected to public or private, the label at the top should change to 'Following' instead of just 'Follow' (Just thought of that now) ..
You could also create a dial (circular)... so 0 is neutral state, then follow public, and follow private.... though I think that the horizontal view is easier to design around.
Multiple modes in UI are bad. So don't strive to place everything in one control. My proposition is on the image. Follow button opens dialog where user could choose an option. You can place some hints and explanations inside so everything will be clear for user.
You should also visualize current following mode. It could be appropriate icon on the Follow button.
How about something like this solution?
download bmml source – Wireframes created with Balsamiq Mockups
The idea is that your simple, single Follow button will change into a segmented button when it you are already in following mode. Clicking on the main part of the button will toggle if you follow at all or not, but in following mode, you can also click on the second segment of the button that will toggle the privacy mode. I do like the feature that the text changes from Following to Unfollow on hover, though you should consider that with increased use of touch devices, hover is not available everywhere.
I think the bottom arrow from Follow to Following publicly is something to think about. I think I would expect that to work if I just (by mistake?) clicked the main button area in public Following mode, so I can easily correct my mistake. But I would also expect that by default, following would be private. So, maybe you should only return to the public Following state if your just came from there, and not if it was longer ago. Say, for as long as you're on the page or something like that.
Both of the solutions you describe have great potential for confusion. Using a simple push button as a toggle is problematic (I know twitter gets away with it, nevertheless...) but a 3 state toggling push button is extremely confusing. And of course on touch UIs the hover change won't be available.
The most clear and easiest to understand is with 2 groups of 2 radio buttons or 1 group of 2 checkboxes:
download bmml source – Wireframes created with Balsamiq Mockups
The 2 check boxes has enough info to be complete, but I might use the 2 groups of 2 radio buttons because it has extra (redundant) info which makes it more clear.
This extra stuff might seem to be not very elegant, but it's really necessary. A certain amount of info is required (an amount definitely lacking in a 3 state button) and it is the challenge of the designer to make that info (or inputs) fit well.
What about adding a tooltip that appears one time after the first instance of a user clicking the "follow" button, advising of how the "three toggle" button works?
For example*:
*This is now on our "tour" for new users (thanks to Zurb Foundation's "Joyride" - http://zurb.com/playground/jquery-joyride-feature-tour-plugin)
