Users Hate These Options - All of Them
- Popup when you are trying to leave the page
How dare you do that, I'll jump out even faster just out of spite and now I won't come back even though I was thinking of it before.
- "Different" back button action
I'm trying to navigate around the site. Maybe I want to double-check something before ordering. Maybe I want to see what else is available before I finish the order but now I'll just leave because you won't let me do my shopping the way I want to.
About the only thing that doesn't, to me (and plenty of people I know), seem quite annoying is to offer an extra discount. If done right, that says "we know you're leaving but here is an actual reason to stay". Almost anything else is somewhere on the scale between mildly annoying and dark/creepy/obnoxious.
People Leaving the Page are Not Always Abandoning
People leave a checkout page for a LOT of different reasons:
Want to get as far as they can to make sure they have all information needed before getting approval for the purchase
Price/product/etc. comparison shopping
Competitors checking your pricing
Search engines or other robots (they stop when they get to a real form submission)
Research on mobile or on work computer but come back to actually buy from a regular computer (or from their personal computer)
Many of these will never buy. Many of them will buy, but it takes them multiple visits (which may not even be "connected" because of private browsing, multiple computers/devices and other reasons). Some of them may buy eventually but, particularly for large purchases, will take a long time to finally complete the transaction.
There are a lot of variables. But one thing is consistent: Users like the near anonymous, flexible, laid back, at their own pace, experience of buying online. Start nagging them and they may just say "I'll go elsewhere", just as they might walk out of a (non-Tesla) car dealer with a pushy salesman.
Let your products, prices, special features (whatever they may be) sell the products. Make the best web site you can - don't bother users if they seem to be going away - if not bothered then they may actually come back when they are ready to buy.
Direct user research - e.g., focus groups, user testing, etc. has definite value. But don't put your anonymous real-world users through a "why are you leaving" interrogation, no matter how small it may seem.