Is there a best practice for laying out buttons with different functionality?
Let's get back to this in just a little.
First, let's look at (emphasis added):
I have four buttons on my interface that all look the same but don't all act the same.
So you've asked a question about layout, but at the core the real issue you're trying to address is how do we create cues for control interaction.
Layout is definitely a key way to denote particular elements of control interaction: most specifically and frequently, semantic relationships to affected processes.
Can layout also be used to address differences in control behavior?
Let's go back to basics.
How do we learn how a control works, functionally?
First, we examine it. What does it look like, what does it remind us of… essentially, what does it seem globally consistent to, visually?
Even if we have no external references to figure out how a control should function, we can still try playing with it. How does interacting with the control change its visual appearance? Does interacting with Control A affect any other controls? Is there a consistent difference in the appearance between the controls that affect the operation of other controls, versus those that seem self contained?
Every time we recognize a pattern, we attempt to re-apply that when interacting with something that seems to meet the same pattern. Essentially, we form a schema and adjust it as needed, usually with a preference for fitting seemingly new things we see into our current schema (assimilation). This is what allows us to quickly allow current models of interaction to apply to scenarios where something appears slightly different but seems to adhere to the overall pattern.
A good example is a toggle switch. We quickly recognize the exclusive state binary behavior of such a switch because it's modeled after switches that are used physically, where if such a switch is toggled in one direction, it physically can't be in the other state. While it would be easy to consider the visual look of most toggle switches to be simply skeuomorphic, they're actually important visual references that are globally consistent and consistent to the physical behavior they are meant to serve as cues for.
Now, let's try something a little more interesting.
What is the difference between a group of radio buttons and a group of checkboxes?
If each group is bounded in a box, how do you, as someone trying to achieve a result, know how each group will interact to that end?
- Global consistency and prior knowledge within similar contexts (software/web): We know how each group will interact because we know from prior control use in other places that a round button denotes linked, mutually exclusive states between it and other buttons in the group, and a square one denotes independent states, even if each one is semantically or otherwise related in some other way (thus the grouping).
- Related (global) knowledge from different contexts: we may have seen a physical device using radio buttons (such as, say, older radios they derive their name from). We may have done tasks that involved physically (pencil/pen) checking off choices or filling in circles (scantron, etc) where similar paradigms are frequently adhered to.
- Discovery through Exploration (learning specific to this interaction, but carried out with expectations from prior learning): when we play with the round switch buttons, selecting a different one de-selects any previously selected. We can infer that this is behavior common to these because they visually appear different from other controls, but appear relatively the same within the same type of control. This behavior doesn't happen with the square switch buttons.
So what happens when all of the buttons are round, but one group acts like radio buttons and the other like checkboxes?
Confusion.
We can no longer rely on our current schema to provide meaningful information, which means we have to test every set of controls to see how it works. There might be cues provided, but the inconsistency with what we already know and how we're already familiar with interacting with the world will cause degrees of frission even then: essentially, whether consciously acknowledged or not, this creates a state of cognitive dissonance that is taxing and frustrating. Even within a common task set (a single form, application, etc) the conflict will remain for quite some time, before the specifically different knowledge can be partitioned to applying exclusively within that task domain.
So what about layout?
Well, the problem is that layout exists at a level wider than that of simply the control itself. Visually, layout is a secondary signifier to the control's action or import, compared to the control's own design.
Differences and similarities in how the control itself looks will always take precedence in how someone interacting with it interprets its anticipated action behavior.
Failure for the control to function in anticipated ways will cause frustration when the control behaves differently from what was expected.
Failure for the control to function in anticipated ways will also cause hesitation throughout the task domain: the person interacting with your software is no longer able to rely on being able to differentiate simply by looking at the appearance of the control itself.
Hesitation is in turn frustrating.
In short, as an answer to the question asked within the question body:
Don't rely on layout to signify different behavior between otherwise visually identical controls
In terms of different control designs, I think a number of the options proposed have promise.
Using a dropdown for a ternary state control is not a bad idea.
Using a connected three way "switch" control is also, in my opinion, not necessarily a bad idea. I would caution that it might become unwieldy depending on the text used.
Using a two way switch control may be problematic, as they are usually expected to be binary: an exclusive or where one must be selected and the other is thereby deselected.
Do use layout to group controls based on what they affect
Placement is helpful for mapping to affected objects not only physically (e.g. placement of controls on a stove), but also at more abstract levels.
Placement and grouping are also helpful in separating out controls from other objects competing for attention, which can be useful in reducing stress of trying to process what otherwise might be neighbor controls for possible relationships in overall functionality.
Slightly more simply, if the immediate concern in what a set of controls affect is limited only to those controls, it can help to separate out those controls either via distance or via visual cues (separating lines, bounding boxes, etc).
Just be careful that you don't accidentally mix cues up with other possible interpretations that create needless confusion (for instance, if placed too close between two controls, a single vertical separating line can sometimes look like it instead denotes a binary switch state for the controls it rests between).
I think layout could be used to your advantage here, but I feel you must first address the appearance of the controls in some way that creates a definite visual cue differentiating those which have disparate action behavior.
If all of these apply to essentially the same object/aspect being affected, just in different ways, I would however advise keeping them grouped together if possible, to retain that significance, especially since it is only 4 (or 3) separate controls.
