Great question, really!
The problem does not lie in Yes or No. But if the action is Positive or Negative.
For example,
"Are you sure to permanently delete System32?"
"Are you sure you want to backup this photo?"
As you can see here, the principle of good design is violated. If you specify Yes or No in the UI of your website/app, you're likely to reuse it. Which causes most of the problem.
Regardless if the event is destructive or not, the User is confused with the Yes and No and might randomly click on one without reading.
A good designer will try to be accurate with the options he provides with the action to be performed.
For example,
"Are you sure to permanently delete System32?"
Delete Folder (Red Text)
Cancel (Accent Color)
"Are you sure you want to backup this photo?"
- Backup to Google Drive.(Green Text)
- Cancel (Accent Color)
The text color is completely optional. However the general rule of color psychology suggests that we all understand green is positive and red is negative.
Now however you place the options, if you have been specific and have stated in the option itself in what it'll do, the positioning will never be a problem.
However, I personally like positive actions to the Right and Negative towards the left.