The most general answer of course is, "it depends." Who are the people using the product? What are they trying to achieve? Are they using it on a desktop or mobile phone? Are they using a mouse, touch screen, or keyboard to navigate (or other assistive technologies)? Is there flexibility for them to increase or reduce the density to what suits them?
Here is an interesting article I found on UX Collective "How white space killed an enterprise app (and why data density matters)" written by Christie Lenneville and Patrick Deuley.
Everything that article says is a good point and says it better than I could say, so I recommend checking it out.
You also need to ensure accessibility for those with vision, mobility, and/or cognitive considerations. Make sure everything is keyboard accessible. Make sure tables are marked up semantically and read correctly by a screen-reader. Make sure color contrast is compliant. Make sure text and content reflows so nothing overlaps or gets cut off when users zoom in or resize their screen. Read up on WCAG 2.2 for specifications on making sure content is readable and accessible no matter how dense or sparse.
And do thorough user interviews/observations and usability testing. If this is a redesign, have the users perform tasks on the previous/current version and time it. Then have them perform the same task in the redesign/prototype and time it. Did it save time? Did it prevent costly errors? Did it improve retention or conversions?
I think the best framework is around the UX research, though I know that doesn't provide the kind of specific answer you were looking for. The most reasonable density is the one that allows people to do complete their task(s) most efficiently and without error, frustration, or the need for home-made workarounds.