Is there a good way to show that you cannot register with a social media account, only login if you already have connected it to your normal account?
The underlying problem seems to be that your design is broken.
The point of providing an option to log in with Google or a social media account is to avoid users having to set up a separate account for your site. Really, you should fix this if possible.
You may need to capture some extra information that isn't part of the user's external account, but you could have a process to capture only that information and not require a duplicate sign-up process.
Of course, I know in the real world it may not be possible to go back and make a fundamental change.
If major changes are not possible, I would suggest:
If people click on "Log in with Google" and enter an account that is not recognized, you could continue to display the log in page, with a message such as:
Google account not recognized. Please log in with your normal account or create a new one to connect it to Google.
Note that I would assume that the user has an existing normal account with your site first (hence displaying this on the log in page rather than going to the registration page). If you assume they need to create a new account in this scenario, you run the risk of duplicate registrations.
It probably also makes sense to direct them immediately to the "connect Google account" functionality once they log in.