You really asked two questions here, but nevertheless:
On question #1
Should a 'Calculator' support '{' (curly braces) and '[' (box braces)?
Yes, Accessibility is the keyword here. Not as in "physically impaired" but as in "learned behaviour". Some people will just want to use curly or box braces, because that's how they're rolling on paper and you want the barrier between real life application and virtual application to be as low as possible.
Users should be able to use interaction patterns they already know (anything else is going to be frustrating), so supporting those braces is giving them more convenience and isn't going to hurt anyone who wants to use parentheses exclusively.
On question #2
Should I optimize my program for a Programmer's approach or a Mathematician's approach?
Why not both? You probably know matching brace highlighting from many IDEs and Editors, where two matching braces are highlighted if your cursor is on one of them.
Another approach is colour-coding, used in Excel as pointed out by stereoactivo.
Both these approaches are applicable and ease readability for both the Programmers and the Mathematicians writing style. This is where your virtual application can top the one in real life, which is always nice.
Other than that, what do you really mean by optimisation? What were your ideas on optimising for one of the two?
On "question" #2.5
[multiple braces] might make processing more difficult (detecting syntax errors, mismatched braces).
That is certainly correct, but you care mostly about the user, not the programmer. (And let's be honest, mismatching braces aren't a though thing to implement. If we're talking about a physical calculator, you should of course consider building cost but I assume your question was meant to consider applications only.)
Also, that is exactly what these additional braces are used for in the first place: detecting errors of thoughts. If you write mismatched braces, you certainly got something twisted in your mind. Giving the user a hint upon that is certainly going to improve their experience.
School of thought to follow / TL:DR;
In general, allowing users to adapt known patterns is something to be desired in every application and known to be the key factor of intuitiveness. Therefore, supporting both writing styles, including checking for matching braces, seems to be the best solution.