What should be the maximum count for a notification badge in a app and what about when its overflow.
I think you need to answer the questions that @graham-herrli asked to really decide what is the correct maximum number of notifications you should show.
About the overflow: an optional implementation is to change the count to a symbol that will represent the overflow.
Example: google chrome on mobile changes the tab count to a laughing emoticon ":D" when you have more than 99 tabs open.
There are no hard rules for the max count display. Gmail shows '99+' and outlook shows the actual count, something like '278'.
It is completely dependent on the kind of design you are proposing and limitation of the page real estate. It could be rare scenario where the numbers of digits exceeds more than 3. It depends on the app you are building and the average notifications per day a user receives.
The below illustration is a reference for badge shape and count only. Probably, you can adopt this wherever required, but again based on your page real estate.