When our sales people asked for a feedback mechanism in our TV app to determine our net promoter score, we went through this thought process as well. We identified a few objectives:
- Don't interrupt anything the user might be doing at the moment.
- Make it trivial to click away/ignore for users that are annoyed.
- Be concise, short and polite.
- Don't produce a "door slam" by showing up too early (we decided to show it randomly, but at most once in 6 months, and not within the first 6 months of use, so the user has had some time to form an opinion of our app)
- Provide some (indirect, minimal) value to the user. In our case, that was a comment field that allowed for free-form feedback, which went to support.
As examples, we never show this window in fullscreen (because it's likely that the user would be watching TV from the couch), I think we don't even show it if a TV window is open and playing back. It is a non-modal window, which allows all other functions in the app to continue uninterrupted (e.g. scheduled recordings). And we check for mouse/keyboard etc. user activity and only show this survey window when the user's been idle for a while, with our application frontmost.
The window has both OK/cancel buttons and a close box, so people can just close or hit the keyboard shortcut to get rid of the window without hassle, keeping the annoyance to a minimum for those people it still annoys.
The advantage of the freeform field was that we got honest replies from users, e.g. where some channel assignments had changed or reception problems were happening in some locales, which we found out about way earlier this way, and managed to fix for most people weeks before we'd usually have become aware of these issues through the usual channels.
So be aware that the default effect of all survey windows is to annoy the user, and actively engineer against that. If you're on the web, you can't necessarily measure user activity, but you can e.g. do something similar by placing your survey at the end of an operation instead of before, and make it easy to ignore (e.g. by keeping the navigation and some useful information around it, so users can continue their work without interruption).