iframes are usually discouraged because their navigation issues. they're also a bit more heavyweight than Ajax (basically it's a mini-window inside a window instead of content), so they require more memory, and load slower.
However, they enable security and distribution of responsibilities.
Navigation is not an issue at you I guess. If users don't navigate to that modal window (facebook enables people to send links of opened modal windows like galleries!) then it can be fine, depends on the two factors below.
If you're also in full-control of the application which is displayed within the iframe, usually an AJAX request is preferred.
However, if you expect the applications to have diverging responsibilities (eg. you want other sites to use the features as well, or you want the visual design and interaction models of the content of the iframe to be independent of the outer site), then iframe seems to be a better option.
- are your users navigating to that view where the state of the iframe is important?
- yes: forget iframes
- no: are both applications yours?
- yes: do you want to keep the same visual layout/interaction model on a number of sites or do you want the content to blend in?
- yes, there'll be multiple sites using this feature, but the interaction/design should be dependent on the feature not the site using it: use iframes
- no, it's only one site using it / interaction is dependent on the site using it / has to blend in: use AJAX instead
- no: use iframes