There is a page, which can be edited by the user, and there are two actions - Save and Cancel.
Would it be appropriate to show some kind of alert, once the user accidentally pressed Cancel after making some changes?
It heavily depends on the damage (or lost time) that the user has to face with the "Accidental Cancel" (or even "Accidental Ok")
If it means that the user would lose more than 10 minutes (or 2 days for some financial transactions), then I would ask for confirmation.
It it means that the user has to fill-in his first name, last name, age again after an accidental cancel, there is not much benefit in confirming.
Trying to configure the behaviour of "Accidental Cancel" will add complexity to the problem. It will still be useful, but only if the user is a power-user who uses the site repeatedly, almost everyday (For ex., if he is an operator of a shopping terminal doing the billing for customers). For a power-user, it is worth to give every single trick in the book to speed up his work.
Short answer - humans are not robots and make mistakes. They click things by accident.
If you can detect that they have made some changes it is often a good idea to display a warning that those changes will be lost.
To make the whole experience cleaner you can include a "don't warn me about this next time" checkbox in the warning message, for those users who don't want to be warned about this if it happens again (this addresses the valid point raised by colmcq).
The error prevention message will make sense if canceling all the changes has huge consequences. Consequences can be in form of lose of valuable time or miss happenings.
For example if they hit cancel they have to re-write the entire form once again from the beginning, considering it's a long form.
Or once hit cancel, the system takes a different course of action leading to loss of money, recourses, more complains, unsatisfied customers etc.
well there is something similar on here when you close a browser window "stay on page or leave" or some-such
However, in your case the user has already made their decision. If they wanted to save their work they would have clicked save. Sure they might click cancel by accident but this would be a minority of users. By having a confirm cancel step you create an extra click for everyone else.
Dont do it