Don't do this.
An unsolicited modal popping up and asking the user a question is just plain annoying, no matter when you do it.
Asking why the user didn't purchase, in particular, is also an annoying question. It will come across as overly aggressive marketing.
This modal would drive customers away.
Also, as your question highlights, there is no good time to pop up the modal. Even the user leaving the page (the most obvious point at which to do it) doesn't mean they have decided not to purchase. Perhaps they plan to come back later. This limits the usefulness of the results, as well as further annoying customers who actually did plan to buy something.
The desire to know why users aren't buying is understandable. Here are a few possible alternative approaches:
- Test different approaches on your site. For example, if you think users might not be buying because prices are high, temporarily lower prices and see what that does to the convert rate. If you think it might be insufficient information, add more information to some products and measure the effect.
- Use web analytics to see what users are looking at and when they leave (as suggested by adamsoh). This can uncover certain kinds of problems. For example, if many users start the checkout process, then abandon it at a certain page, it may indicate a site design problem.
- Have test users evaluate your site and provide feedback. Again, this can uncover certain types of problems, but not all. It might highlight that users don't understand how to do something on your site, for example, but if the problem is price it probably wouldn't uncover this, since the test users aren't making real purchases.
- Investigate competitors. This is kind of obvious, but: when you search other sites for the same product, do you find lower prices? Do their sites look nicer, have more information than yours, etc.? Put yourself in the customer's shoes and try to consider why someone might prefer another seller to you.
- Provide a more voluntary method of taking a survey. For example, have a button on your site that offers a user an incentive (such as a chance to win something) if they agree to take a survey. Collect email address and send the survey later, when you are sure whether or not they purchased anything.