My devs are currently working on a form that appears in a modal dialog- it was pretty big to begin with, but now they've added some fields and it's hanging off the screen- now they want to adjust the css to compact the form, which will leave it mismatched with the rest of the application. The reason it's been implemented as a modal dialog is so the user won't lose their place in the search results after clicking on one of them to edit, so I can see they have a valid reason for wanting to do it this way, still, having a modal window so large it either needs to have it's styles squashed or some kind of scroll implemented just doesn't sit well with me either. How big is too big when it comes to modal dialog forms? Should I have them split it into a multi-part form or just squash everything like the engineers want??
Functional excess is something you deal with on daily basis, especially as the project gets closer to something I call 'climax' (when Client gets wild and wants to put everything s/he did not consider before - meaning 'additional requirements' rather than errors at the phase of analyzing and planning functionality).
Regarding modal size itself, I would say you get too close to "too big" when any of these situations occur:
But don't worry, you can balance the above, by e.g. mixing some trickery, for example:
I would do this in this order. If any of these is not be possible to implement, I would skip to another one; if applying one is not enough, I would move to another one as well, to improve the solution.
I think Trello has an interesting solution to this which I have copied to one of my own projects. You can scroll the modal without affecting the scrolling of the interface in the background.
The width and position of the modal still lets you see whats behind it which gives you a sense of context.
This way you can keep the style of your form elements consistent without squashing anything in.