Popup windows can be classified into three distinct categories based on their modality:
- System Modal windows (lock any interaction with OS)
- Application Modal windows (lock any interaction with application)
- Non Modal windows (allow users to ignore and continue working)
1. System Modals:
This is when the whole OS is locked and the user must select an option in order to continue. This is the type of popup that you want to avoid; unless you have a very important reason to deal with it.
If you are on the web, as far as I know, you will not deal with it. Btw on Windows OS a typical system modal (also has an overlay), looks like the following:

2. Application Modals:
This is the interesting part. Now pop-up blocks any further work in the application window (that showed the popup), but let the user to interact with other applications.
In a browser this could be a javascript generated alert box, or anything else like a browser settings dialog. For example this is from chrome settings somewhere.
App doesn't let you click anywhere else in its window, and also uses a shadow instead of black background.

When using alert boxes, have in mind that users can disable popups in browser, so it's a bit risky to rely on them.
3. Non Modals
Those type of windows allow users to ignore them, and continue doing what they were doing. Probably they can't relate to your question.
But what about overlays?
Below you can see an image gallery . It is a popup window and it uses an overlay (like facebook does).

The interesting part here is that the black overlay doesn't indicate that something is locked; it is just decluttering and brings contrast to the field. Also when you click on background, you are leaving the "gallery mode"!
Now if we compare image gallery with application & system modals, we can easily observe a pattern (transparent background and a window), but their behaviour is somewhat different.
And as the time passes, users on web expect that they can easily close windows and continue what they were doing, just by clicking a transparent black background.
So we must be careful when a user has to take an action on a modal and you want to indicate that something is unavailable to click.
By the way for sake of simplicity i used the term window equally to dialog, which is not always true