To your specific application
Given that your app is vertical only, AND that the notch form factor may be an issue, upside down is probably lower down on your list of requirements. If your UI was such that the bottom of the screen was generally pretty open anyway, then it's not such an issue.
Consideration of how the app will be used
The environment and setting that the user will use the app should be a consideration. It could be as simple as the user being half asleep and reaching for the phone on the nightstand. Or they may have to grab the phone and use the app when distracted or only able to use one hand. Supporting alternate orientation in these situations might be important, or greatly decrease user frustration. A half asleep user may be frustrated that the app can't handle upside down for example.
Applications where orientation is an issue
The only kind of app that could not support it would be an app that had to be oriented vertically (did not support horizontal mode), and where a screen notch would interfere the UI when upside down. Another example of interference might be a camera or other physical aspect of the phone being obscured by the user's hand or difficult to access when holding the phone upside down.
Potential for user frustration by not supporting it
But unless the above is true, then I believe upside down should be allowed. Not allowing it just creates a small element of user frustration and bad user experience when the user turns their phone and the app does not respond.
On apps being able to adjust for orientation
If orientation could cause a problem (such as an upside down picture from the camera), then the app author should account for this. Apps that allow horizontal orientation don't have a right and wrong side and can properly handle photographs for example, so there is no reason why an app couldn't correct for an upside down phone.
There might also be cases related to integrated sensors, such as a compass app. But even then, this could theoretically be corrected for.