Color Separation
I don’t know how much help this is, but I’ll re-cast your question into something more specific. As you intuit, you want high color contrast between unrelated things, and low color contrast among closely related things. So your question becomes, how do you determine the perceived degree of color contrast between any colors?
The International Commission on Illumination (CIE) long ago established equations for calculating color contrast of illuminated displays, and these equations are still largely correct for modern displays. The procedure is to convert your RGB color values to CIE Luv color space coordinates, which specify the perceived brightness (L), hue (flat angle from L axis), and saturation (distance from L axis). The degree of perceived color contrast between two colors is equal to the Euclidean distance between the two sets of Luv coordinates.
Here’re the Luv equations for the sRGB color model, which is a generally suitable estimation for typical consumer-grade displays:
Convert your RGB values into “rgb” values:
r = (R/255)^2.2
g = (G/255)^2.2
b = (B/255)^2.2
Matrix-multiply your rgb values by a constant matrix to convert them to XYZ color coordinates. As three formulas (rounded to four decimal places), it’s:
X = 0.4124*r + 0.3576*g + 0.1805*b
Y = 0.2126*r + 0.7151*g + 0.0721*b
Z = 0.0193*r + 0.1192*g + 0.9505*b
Calculate L, the gray-scale brightness level:
IF Y/0.9999 > 216/24389, L = (Y/0.9999)^(1/3) * 116 – 16
ELSE, L =24389*Y/0.9999/27
Calculate u’ and v’ for the two hue dimensions (if X, Y, and Z are all 0, then set u’ and v’ to 0):
u’ = 4 * X / (X + 15*Y + 3*Z)
v’ = 9 * Y / (X + 15*Y + 3*Z)
Convert u’ and v’ into u and v coordinates.
u = 13 * L * (u’ – 0.1978)
v = 13 * L * (v’ – 0.4683)
The perceived color difference, D, between two colors (L1 u1 v1 versus L2 u2 v2) is found with the three-dimensional Pythagorean equation:
D = sqrt[ (L1 - L2)^2 + (u1 - u2)^2 + (v1 - v2)^2 ]
As a rule of thumb, for indoor use of consumer displays, a color difference of 100 is a whole lot and 20 is a very little.
There is a vast range of brightness, hue, and saturation differences that achieve the same color difference. The good news is this leaves you lots of latitude for choosing your exact colors, so you can achieve other user experience goals. The bad news is this leaves you lots of latitude for choosing your colors, so you’re pretty much on your own.
I’ve more on color use, with examples and spreadsheets at Breaking the Color Code. Bruce Lindbloom provides on-line calculators and more on color theory.
Non-color Separation
All that said, I think you should consider that maybe color choice isn’t the issue. Generally, color, especially the hue and saturation components, are not that great for indicating levels or a hierarchy. At the very least, you want it to be redundant with some other graphic dimension. For example:
Spatial Position. User will tend to assume information from the upper left to the lower right are ordered from general to specific. Content close together and separated from other content by white space will be seen as similar. Indented content is “under” outdented content.
Line. If there isn’t space for white space, lines and boxes may be used to show separation and nesting.
Size. Bigger things, especially headers in bigger font, indicate higher levels in a hierarchy.
Frankly, your current use of these elements in your wireframe looks sufficient to me –even in its current grayscale form. Using color to further distinguish the levels may be over-doing it, resulting in a cluttered-looking design. Why do you think you need color contrast to distinguish the hierarchy? Have you tested this design? Exactly what problems are you seeing? Maybe color isn’t the solution to your problem. Maybe you should use color for something else.
