The answer to your question is very simple, you can't. If it's not public, you can't study it. Not without breaking some rules, which is not recommended.
But if you still want to do the study, you still can, you just have to change what you are studying.
First, remember that for doing the study, you need somebody that know how to do that and you need mockups, wireframes or any other kind of element to run the study over.
Once you decide to launch a site, you most probably, have some knowledge of the area of expertise that the site is going to cover, plus you have knowledge of interactions and interfaces that you have seen, so you can combine both. For the interaction, you have to replicate aspects of other sites that have what you want and study that part with your users. Of course, you can substitute what you want for what the competition offers. Even if most of their site is private, you still have access to colour schemas, positioning, flexibility od the design, reaction to different screens, technology used, loading times, etc. Use all that.
If the site that you want to study offers the option to sign in, and is something reasonable to do, do it, and use what you can see.
Of course, we are talking about legal uses here, just studying and analysing.
But, after some point, you are going to be on your own, there is going to be aspects that yuo can not see, either because you can't register, or because you can't access a section of the site, etc. That part may happen to anybody, so don't worry too much about the aspects that you can't see.
Finally, remember that the people that know how to do such a study, can recommend you options for the elements that you can't see, they may even have designed sites like the one you want, so ask as much as you can, and listen to what they say.
About your specific question of breaking rules, that is something that has to be answered on a per site basis. It's very strange to have something really new and original, so most probably whatever a site has, is also somewhere else, just try to find it.