I'm interpreting the question that you have a custom software product which you are developing, that offers some functionality with and some without logging in.
You could provide either a chat window (e.g. in a discreet popup element) offering a possibility to talk to the product team or a questionnaire to tell about the motivation for using the product without logging in.
If there are functions that would identify the user albeit not logging in - most likely user inputting their email address at some point - you could contact the users directly for feedback.
If you know there are some problems with the sign-up page, one approach would be to provide some tips in the difficult phases, or offering the aforementioned chat support on that page, too.
Of course, if and when you know what the problems are, just fix them and iterate.