See title, this needs to be at least 15 characters
If I'd asked people what they wanted, they would have told me 'Faster Horses'. -Henry Ford
That aside, try not to think in terms of "what is best for" and instead use "what are the goals of" the user. The user may not know that there is a better solution to their problem. He/she may be "used to" a solution they know without realizing that there exists another solution that is faster/cheaper/easier/etc...
If you're designing something that people are going to come to, use briefly, and then probably not use again for weeks or months... that's not a good opportunity for showing them what's "best". But if you're designing something that people are going to live and breathe and use every day—like a calendaring tool or a software development IDE—that may be a time to try and re-train them on a novel paradigm.
A while back I read Joel's "User Interface Design for Programmers" and he drew a distinction between learnability and usability. I rephrased it like this:
If you’re designing emergency exits on airplane, you need to make sure that someone can get the door open without a tutorial. The flight staff is required by law to explain how to work it, but it still needs to be obvious. A person looking at a picture of the exit must be able to accurately answer how it would work without even touching it.
Yet most software doesn’t have this life-or-death aspect. So it’s acceptable to break a couple of initial expectations or stereotypes that were learned in other programs…if your software’s new method has a productivity payoff for the relevant task. The book describes this an interface which is “learnable”—as opposed to one that is immediately “usable”.
My opinion is that developers should be daring and create "learnable" interfaces instead of just imminently "usable" ones if there is a great benefit to doing so. I note in my article that we have a lot of tools for teaching that weren't so easy to do in the past... like screencasts and interactive demos. Yet I'd be careful to use metrics to justify one's claims of benefits/betterness.
"What's the difference between what's best for the user and what the user is used to?"
I don't believe you can separate them so distinctly. Users frequently want systems that match their mental models even if that means a slower or less elegant (etc) approach to the task.
Also, I'm leery about saying that I know "what is best" for the user as the question implies. After preliminary analysis I'll design multiple options and test them to see what users want/prefer. It sucks when they choose something I hate, but they use the system - not me.
I presume that you are talking about 'best for' in some idealised abstract way and what the user is 'used to' as some kind of latency and resistance to novelty. They are not mutually exclusive. There are examples from all over usability and ergonomics where users do not do what is 'best for' them as they have gotten used to doing things in a certain way. This is the basis of interface metaphores; to apply a representation that the user is 'used to' in order to help them get to grips with a thing they might not otherwise understand. What is 'best for' the user (in terms of efficiency and situational awareness) is if they learn the complexities of a system uninhindered by metaphore, but this learning may require extensive effort on the part of the novice user, presenting a learning curve which is too steep for most, leaving the only possible users as the expert elite. So what is 'best for' most users (in terms of ease of use) is to give them familiar hooks to allow them a way of understanding which will get them using the parts of a system they need. Now any new system will have to take into account the modes of thinking and behaving held by existing users, as well as general modes of thinking and behaving. What is 'best for' users then becomes very closely related to what they are 'used to'. When I first read your question I immediately thought of the classic QWERTY vs DVORAK question? Which is 'best for' typists?
Strive to achieve designs which either conform to the user's expectations, or are so dramatically better that the user will be happy to make the switch. We are all resistant to change. Don't subject your users to change for minor improvement; they won't like it.
If you don't have a MUCH BETTER way to do it then stick with conventions. You don't want your users to have to relearn how to use your system if they are familiar with conventions from other systems.
However, if you have a way to do it MUCH BETTER and they have to do a little learning early on but long term they will thank you for it. Then do it.
QWERTY vs DVORAK is an interesting analogy. I'm actually a DVORAK user. I could touch type on Qwerty but I knew I would probably be typing for another 50 years so I personally decided to buck convention and try a better way. Turns out it's not that much better really when the convention on every other system including my phone is Qwerty. On my machines I'm comfortable and efficient, and my thumbs can still do Qwerty just fine. But I can't make the mental shift back to touch type Qwerty on the keyboard. It really wasn't worth it to leave what was conventional in this case.
Jonathan Grudin's "The Case Against User Interface Consistency" (1989) sheds some light on the issue. Especially, I'd never thought about the distinction between external and internal consistency.
... [O]ne must be able to identify good consistency, since foolish consistency is possible. (ibid., p. 1164)
Also note that there is a schism between ease of learning and ease of use. In essence:
... people buy systems and applications not to learn them, but to use them. (ibid., p. 1166)
"If you asked people in 1983 what they wanted in a Word Processor, they'd rattle off the spec for 'WordPerfect'" -- Steve Jobs