You are correct that it does help for users to be able to change their vote for a number of reasons; they could have misread the post, changed their mind, accidentally clicked the wrong one, etc. Perhaps you upvoted a post saying "I want world peace" and the the poster edited it to say "I like My Little Pony", if you can't take your vote back all your friends will think you're a Bronie.
However, you should also impose a limit to the amount of time they have to do so. If a user gets downvoted by someone on the site they may go out with a vengeance and take back every upvote they ever gave them. Or on a Q&A site like this if the asker accepts an answer other than your own you may try to retract your upvote out of spite. If the post hasn't changed there is little good reason to allow a user to change their opinion. That is why StackExchange gives you a 5 minute grace period to change your answer, after that it is locked in unless an edit is made to the question. If you try to change your vote you get an error like this: