I have two classes (say, A and B) of objects represented by two different colors. Now certain objects could belong to both A and B. How should I choose a color scheme for these classes so that I can represent objects with classes A, B and also A-B together, so that the users understand the concept intuitively? Is there any prior work that I could read about on this?
It's hard to answer without knowing more details about the nature of the objects, how they're presented, and how users interact with them. There are a lot of options.
The most generally applicable solution, I think, would be to create a small set of icons which represent set membership, which you could then place next to your objects. I'm imagining flat little chiclet shapes. The icon for dual membership would be half-n-half one color the other.
I don't recommend using a gradient! A gradient implies a range of possibilities, like that an object might 25% A and 75% B. Just use a sharp divide, like those lovely cosmopolitan ice cream sandwiches.
You could probably get better answers with more information about the objects, though. What are they, and what makes them be A or B or both?
You have two classes
What defines each class I'm assuming is a set of defining factors, however you are saying that there are "items" which may belong to both classes because they contain the qualities of both.
In this case for clarity I would create a 3rd class that is a mix between the attributes of class A and class B. you could do this by taking the colors and blending them together or putting class A's color as half with an = sign in the middle with class B's color on the other half. Visually it must be different from both class A & B so that it's easily understandable that it is a mix.