I have done some research on this subject and find much about the "myth of the fold," why infinite scrolling is not great for ecommerce, and how users benefit from a button to click to load more content. I have not found that magic answer for how infinite is too infinite?
My personal experience leads me to believe that you have indeed found the answer already, namely that there is no hard-and-fast rule to follow. It depends entirely on the type of content being presented and the target audience.
The general rule of thumb (just guidance mind you) is that users tend to like to have both options available to them so their experience can be tailored to their browsing style. In many of the online journals/blogs/magazines/etc. the content is often presented in web-paginated views, meaning I have to click some sort of "Next" or page-numbered button to move to the next page.
However they also often provide a "printable view" or (even better) some kind of "see all pages" view. That way if I don't want my reading experience artificially interrupted, I can skip it altogether. I find that I enjoy this most when reading content that requires focus, because I don't have to break my concentration to pay attention to navigation controls.
The key is this: know your content and know your audience. That will give you a firm idea of how you should (or should not) segregate content.
There isn't "a" correct answer. Depends on what your site's goals are. Depends on what goals the people using your site have.
Think about what you want to achieve. Think about the assumptions you are making that are pushing you towards a infinite scrolling solution. Test those assumptions. If they work out well - go for infinite scrolling. If they don't - don't.
There's a nice presentation from the folk at Etsy that takes apart their introduction of infinite scrolling and why it didn't work. Might be worth a read http://danwin.com/2013/01/infinite-scroll-fail-etsy/
how infinite is too infinite?
There's no hard limit. As the page grows it takes more memory and that can cause the system to slow (virtual memory swapping). You don't know what type of system the user is viewing your site on, so a user with a small memory system (an old desktop or a low end tablet) might reach the point of performance problems quickly. This can be a major issue on an ecommerce site if you can't limit page size.
Even if the memory limits aren't stressed to the point of performance problems, an extremely large page is unwieldy.