In general, I use the following guidelines for using modals:
Is it focused? Every time you throw a modal in front of a user, you're disrupting their workflow. Disruption isn't always bad. Sometimes that's what you want. But you have to realize you're doing that and use it for your benefit. Items within a modal should self-contained. A good rule of thumb is that a modal should be used only when it's content is focused or can be shown on it's own page. Examples of this are Pinterest cards, Trello pins, or Behance posts. Notice that they're all self-contained posts, which can be deep-linked to, but all use modals in-line to focus the user's attention on one specific item. The user doesn't have to remember other items in order to interact with it.
Is it consistent? Without clear guidelines for when and how items like modals can be used, they can lose their effectiveness because they'll be used all over the place. Unfortunately a poor example of this is right here at Stack Exchange. On your user profile, your top-level menu contains links to "Edit • Privileges • Preferences • Flair • Apps", all of which take you other pages. The last item in that list, "My Logins", opens in a modal. This isn't consistent with how modals are used elsewhere to alert the user or help them focus on a task.
Is it overly complex? Using a modal on top of another modal is a big red flag that something broke down in your workflow. If your workflow requires a second modal, you probably have some bigger issues with complexity within your modal. Remember: some of the best use cases for modals are focused, simple content. Squarespace use multiple modals for when you add certain content. And while I love the product, this is the item I hate the most in the product. It's confusing and tedious.
Modals are a great tool. Their overuse can blunt their effectiveness in your workflow, though.