First off: no, I don't think there's any definitive research, because what you should do is dependent on the context of your design. Who are your users? What are you asking them to do in a modal overlay? Is it compelling to them or merely an interruption?
Modal overlays are an extremely disruptive design pattern. Abuse of these is rampant, especially when people use them to do things like ask users to sign up for a newsletter or share content on social networks, when all they wanted to do was read a page.
TL;DR: The right thing for the user is to provide multiple options to close an overlay, unless the user must make a choice. If you want to strongly encourage users to make a choice, you can encourage them to do so by not including ESC or "click outside to close" as options, and only including an X. I would not include ESC or click outside to close on a selective basis, and rarely.
That is a pattern I'm using at Wikipedia. We use an extension called Guided Tours which includes all of these close options, some of which can be disabled. (It's originally based on the Guiders.js library from Optimizely, if you're curious.) Some of these are overlay steps, and all of them are little modals that can attach to elements on the page like a tooltip.
In most cases, dismissal rates for tour steps that are not overlays are quite low. For example, since April 1st UTC time, one tour which shows people how to edit had about 800 users see the first modal. Only nine users (~1%) clicked X to hide that step. However, a modal overlay that precedes that step includes an X but disables ESC or click outside to close. It has a 60% dismissal rate. We went with this because it was still outperforming a version where we sent users to a static page and gave them a non-modal CTA. That had a 30-34% acceptance rate, as opposed to 40% in the modal version. I'd say that for a disruptive design like a modal overlay, 40% acceptance rate is pretty darn good, and that's partly because we strike a balance between the number of close options and asking the user to make a choice. In that overlay, we not only include an X, but we include a text link that says, "No thanks, maybe later" to dismiss.