I create a math game where the user has to solve an equation. If the user is right, the application plays sound A. If he's wrong, it plays sound B.
What criteria should I use to choose sounds for A and B?
When the student gets an answer wrong, you probably shouldn't play any sound at all. From About Face 3, chapter 25:
Given the choice of no noise versus noise for negative feedback, people will choose the former. Given the choice of no noise versus soft and pleasant noises for positive feedback, however, people will choose the feedback.
The chapter gives a few examples of how outside of software, sound usually indicates success [and silence indicates the absence of success]:
If you don't play a sound when the answer is wrong, your options for sound A are wide open. I don't have advice on what makes a good positive sound, but I imagine the choice will be easier if you're not trying to compliment it with a negative sound.
RETRACTION: Having given it some thought, I don't think this is the right answer to the question. The feedback Alan Cooper, et. al. describe is an acknowledgement that input was received and the software knows what to do with it. That's not the same as feedback about whether an answer was correct or incorrect. The latter is clearly about the user's success. The former is arguably about the computer's success.
Since I've edited the answer you can revoke your upvote.
Many thanks to @DJClayworth for pointing out my mistake.
Depends on the look and feel of the game and the game mechanics, but you should try and match the sounds to the feel of the graphics and pace you want to achieve, how rewarded a person should feel for answering a single question, versus passing a level, etc.
Examples: if you use avatars(illustrations of people), record someone saying "Correct!" or "Incorrect." Or audience sounds such as clapping, cheering or gasping.
You need to beware of thinking of a wrong answer as a "user error". Your application's purpose is to tell the user whether their entered answer is correct or not. An incorrect answer is not an "error" as such, and shouldn't be treated like "invalid input". It's valid input that happens to be the wrong answer. (Carried to extremes, applying 'error handling' logic would say that you should simply prevent the user from entering wrong answers, which obviously defeats the point).
Games, such as TV quizzes, don't subtly indicate wrong answers - and they don't try to 'guide' the user away from wrong answers, as a good UI should guide you away from errors. Instead they make a fun event out of it, playing sad music, or a buzzer, or a raspberry. Not as fun as a correct answer of course, but still fun. You'll need to choose something appropriate to the age group, of course, and use something original.
It's a different matter of course to indicate an INVALID answer - e.g. trying to use letter keys to enter a number. There its a user error, and you should apply all the normal rules.
You might find the sounds used by Slot Machines to be useful.
IIRC, Casino slot machines only give audible feedback for success - failure is silent. And, all of the beeps and tones and melodies are played in the key of G Major, so that all of the sounds of all the machines in the same space merge together in a way that doesn't clash.
So, perhaps some kind of major-chord melody (just a few notes) for success, and silence for failure.