Many elements, such as light-boxes, will close if you detect a click outside of your box. One simple way to do that: toggle a bool on mouseenter and mouseleave and then on click do
if(!InPanel) closePane(); This is my primary suggestion.
I think depending on the purpose of the panel, the user will sometimes want to scroll down the page to view something while changing settings in the panel. Especially if you've got some sort of live search filter and things like that. If this is not the case I think it is fine to close on scrolling - but again not if the user is inside the panel with their mouse.
While a close button can be a slight nuisance, if you make it an arrow icon inside of the border, it usually goes over pretty well to expand/collapse this way.