In my experience, modal windows are best used to present clear interactions that the user either:
- needs to do (e.g. resolve an alert)
- has chosen to do (e.g. open a photo gallery).
Resolving that interaction should close the modal, and there should already be a control in place that does that.
This is because modal windows interrupt the user flow and break standard browser controls and idioms; you cannot deep link into a modal, you cannot back out of a modal, web spiders can't read modals, they're just not a good place to put actual content.
With that out of the way, I don't actually see any problem with changing the behavior of the browser back button. A lot of websites modify the back button to return to the last semantic "page" that the user was on (c.f. gmail). In this case, the modal is a seperate space that contains content that isn't in the main page, it's definitely a semantic page all of its own, so it makes sense for the back button to revert out of it.
In the worst case, all you're going to do is cause the user to have to click back twice to get their expected behaviour, and in the best case, you're going to prevent a user pressing the back button and ending up on an unexpected, and completely different website (i.e. back to google).