It seems like the current UX pattern is to have both files open at once. If you can only have one file open at a time, should you open the new file or keep the original file open?
It's probably safe to assume that the user wants to start working on the duplicate file right away, but you'll need to be very careful with the experience of not forcing the user to close the other file or force save unsaved changes. They might need an interstitial modal giving them a chance to cancel. If possible, you might want to press on your product team to consider letting the user keep both documents open. Users also like to cross-reference original and new versions as well.
Apart from the buttons here, user's main aim is to create a "new" file that doesn't exists before.
Therefore the underlying aim here is to rename, re-configure or making arrangements on that specific file after restoring it. So that, letting user to be able to do that is more proper from this point of view which means
opening new file at first hand.
In the meantime, before doing this (especially if you're forced to close one of these files), of course you should ask for saving changes first to the existing file and then to the duplicate one (or saved as new).
By this way you may safely onboard users to the new file creating process.
The common term for keeping only the original open seems to be "Save (a) copy (as)". I checked a few applications with this option (Matlab, Inkscape, GIMP, Ristretto).
Your wording is the opposite. Duplicate generally focusses on the new copy. Examples include slides in PowerPoint, and one that's particularly pertinent as it doesn't support opening multiple files: Strava's route builder (web service for planning cycling/running routes)*. Here duplicating a route creates a new one with all the details of the original, and immediately opens the save dialog. This makes it clear that you're saving the new one, and means that the baseline at the moment of divergence is saved (helpful when working online)
* I wish it had a "Save as" or "Save copy as" in addition, but that's another story.
Considering your situation, there're two ways:
Force auto-save the previous/original version and then open the new file.
Create the new file in the background, keep the original file open; however, show them a popup about opening up the new file. If the user clicks on that popup, crosscheck with them if they want to save the original file or not; then, open the new one.
Well, in general, it totally depends on you or your platform actually; there's no strict grammatical rule or something, about doing one or the other. Meanwhile, there's no conventional way either. As a result, you may just follow one, that suits you; and, educate your current users.