What exactly is a modal dialog window and when should you use it? Are there certain cases where you should refrain from using one?
A modal dialog is a window that forces the user to interact with it before they can go back to using the parent application. A great example of this would be a prompt for saving, or the "open file" dialog.
They are often used when a user is forced to make an important decision. Say you were working on a document in Microsoft Word, and you chose to exit Word before saving. A modal dialog would pop up and ask you to either Save, Don't Save, or Cancel. Until you make your decision you cannot use the application, and it will not close.
I agree with Matt that a file dialog should be a modal dialog.
Modal dialogs should be used when the user takes an action that needs additional information to be completed correctly. The print dialog, is also a good example. Or the 'options' dialogs in loads of programs.
When to use modal dialogs is a bit tricky to define, but to me it should be used sparingly, and avoided whenever possible. Look at Fraser's example of the Facebook image loader... that's not necessary as it can be done asynchronous.
Rule of thumb: whenever additional info is needed to complete an action.
Matt already said what a modal dialog is.
As to when it should be used: The rule of thumb is "When no other action makes sense until the user completes that dialog". Read carefully, it does not talk about what you wish the user to do! Many dialogs in Mac are not modal where their counterparts in Windows are modal. This is not to say that Mac is better, but to say that many dialog designers needlessly force the user to complete an action that the user might want to postpone until she is ready.
Examples: 1) a SSH program requires the password to connect to the server. That may be implemented via a modal dialog, though you could argue that opening the options configuration makes sense whether the user is connected or not. A judgement call really.
2) A multi-tabbed browser should not implement HTTP 401 basic authentication via a modal dialog. A user may well want to open another tab to read the password from an activation email.
3) A print dialog in a multi-document interface should not be modal. If the user wants to print two documents, one with 1000 pages and one with 5, but wrongly starts with the 1000 page document, you should not require her to cancel the dialog (and all settings she gave) just so she can print the smaller job first.
Matt answered the question as to what it is. As to when to use it, it's useful when you need the user to focus on one task and one task alone. On the web, a modal is often centered on the screen with the rest of the page being grayed to focus attention on itself and visually imply that the background won't be used until you complete this modal task.
Where they shouldn't be used is somewhat highly dependent on context of your particular project. Where I've seen it abused is when it's used as a replacement for a normal page flow. Example where a user may trigger a task on the main page to launch a modal that then makes an AJAX form to load a form that then steps through 3 modal screens before finishing. In that example, sending the user to a separate page was probably the better option.
The modal dialog is a secondary window that opens on the main one. In order to proceed further with a set of task, the user has to interact with it in order to return back to main window.
A modal dialog window is like a popup except it occurs within the browser window unlike a popup which "pops out" of the screen. A modal dialog window also traditionally have an overlay between itself and the content of the web page in order to give the modal window more prominence.
They can be used for a variety of different things but usually when the user will want to return quickly to the page the modal window came from.
See below for an example of Facebook's style modal window.