Your decision is going to depend on the functionality of your instant messenger and the features it supports. If you are supporting features like video or face chat or voice calling you will need to decide how many users your server can handle at any instant in a single conversation before it breaks down. You will also need to consider the case of how much visual/audio based information users can handle at an instant and if that correlates with how much your server can handle before it starts breaking up (i.e. it starts dropping calls or the video starts to stutter).
If you are just handling text, you will need to look at how many instant messages anyone can keep track of at any instant while in a fast moving environment.
I don't have any research to back this but I believe the google hangout limit of 9 was enforced upon the "principle" of seven plus or minus two items at any instant.
However as Ben pointed out, if you are just looking at a presenter model then your limits are obviously higher, although you will need to look at incorporating a moderator model as well to prevent the conversation from getting out of hand.