First, you should generally be thinking of a percentage of the availible space for a maximum, rather than a set number of pixels. For a minimum, pixel count is more understandable as there may be a minimum amount of data you need to present.
As for how big is too big, it all depends on what you're doing. If your modal requires any sort of information on the "lower" pane, the modal should ideally be as small as possible and movable, a good example of this is the "post tweet" modal on Twitter's web app.
Modals are generally supposed to be small, focused tasks where other information is unnecessary; confirmation dialogs, log in screens and similar. As such they should almost always be as small as possible.
I do however disagree that modals should never ever cover most of the screen; the Print modal in Google Chrome is a great example of a full-screen modal. The Print Screen shows a full preview of how the page will print and gives a sidebar full of options; in this case being able to see the main page is more of a distraction than anything. This is a pretty rare case, and generally modals should be much less than all of the screen, but it can make sense.
If you do need to make an almost full screen modal, do consider if it should be it's own page rather than a modal; in the Chrome example it makes sense that it's a modal. Chrome's Print dialog is completely tied to the current page. If this is not the case and instead the modal is a completely separate and complex page it may need it's own page rather than a second level modal. But it's hard to recommend which is better without more specific details of your use case.
