Firstly, it's a given that whatever you do, you need to test it with your users. Whatever we suggest is taken to be a good first start.
That said, I think that YouTube's design is not very good. As you rightly point out, people usually click on the up or down vote icons near the counter - which is to be expected. The counter indicates the current voting, and so that is usually where people's attention will be when they are thinking about voting.
It's generally better to have the voting buttons very close to the counter for the voting. This site is an example of a counter with good UX.

I suspect that the reason that it is separated in YouTube's case is related to a legacy consideration. I for one, would not suggest a new site follow their example as it is unclear and unnecessarily duplicates the up and down vote icons.