"This name already exists" is quite bad. The fact that the name exists is no problem. The problem is that a file with that name exists. So of the three initial options, "File name already used" is the only one that is correct, the other two are plain wrong. It's still not good: The problem is not that the name was used for a file at some point in the past, the problem is that it is used right now. So better would be "File name already in use".
Then come three variations of the question "do you want to enter a different name". I don't know what the situation is exactly if the user insists on this name: Will an existing file overwritten, or will the operation not work?
If the operation doesn't work, then it is pointless to ask whether the user wants to enter a different name - the one they entered isn't going to work. If the user definitely doesn't want to enter a different name, that's what the "Cancel" button is for. Apart from that, if I'm asked "do you want to enter a different name", I'd expect a way to pick YES or NO.
Most likely you want to encourage the user to enter a different name. If that's what you want, that's what you should say: "Please enter a different file name".