In websites that have blog posts, if I look at the right scrollbar to get an idea of how long the post is, it is often deceptive since a massive amount of space on the site is taken up by comments. What would be a good solution to this that wouldn't frustrate users who did want to see comments? Perhaps display 10 most recent/popular as a default and require the users to click something that said "View all comments" to unfold the accordion? I've seen this on some big news sites, but generally design blogs don't follow this convention.
Good question. First each user have different perspectives. In most of the technical blogs comments are as equally important as the blog itself.It is not the same case that all the users dislike the long page functionality that has the entire comments. May be there are some just like you.
If you just want to show the exact page length to the user and not deceive him with all the comments space, Just leave a limited space like five or ten comments and then keep a horizontal Load more or Show all tab where the user can externally click it to view the rest of the comments.
You can also make it real user friendly by just loading more comments when the user scrolls down the end of the page (ex:Facebook). It is also a great way to enhance performance since it is an on-demand loading.
Comments tend to be important for users who visit blogs/news/reviews sites, so dedicating a lot of real estate to comments is good design for these sites, maybe not aesthetically in your view but definitely in terms of meeting the user's goals.
I'm fond of websites that don't load the comments initially, so the page size is an accurate portrayal of the article length. You can then either load the rest of the comments automatically when the user scrolls to the end of the article, or have a button they can click. This will also help with load times on sites with heavy comments.
Everyone here has already mentioned limiting the number of comments. As an alternative, you could opt for a reading position indicator.
What I mean by that:
Notice the red bar at the top. It indicates how much of the article you've read so far.
A variation on this could be, just showing how high each part (article and comments) is relative to each other. You could of course still add a position indicator to it if you like.