What's the difference?
For those who don't know, popup is normally a child window opened by a webpage that blocks any interaction with the parent window (until the popup is closed). A modal is not a window, but rather an HTML construct that would give an impression of the window, and would block any interaction with the rest of the page.
Here's how a bootstrap modal looks like:
Source: http://v4-alpha.getbootstrap.com/components/modal/
Which one to choose?
You should most certainly choose modal over popup window in any realistic requirement.
Popups have a tendency to get annoying to users, especially because they would often prevent users from going back to the previous page or even navigating to other pages (depending on the way the web developer has configured them). Modals on the other hand are the way to go for better animations, ease of use and less annoyance.
Another problem with use of popup stems from the popup blocker configurations. If a user has blocked popups on your website once, chances are you will never realize that, and they would think that your site is not working. Modals can not be disabled, and thus give you complete control of how users see your site.