When you are removing text as a visual cue for a user you are taking away a very clear way of communicating what a control does. Because of this there are two things that you should consider when designing your icons:
Firstly you need to keep control over your metaphors. Interactions in software that are similar to real world interactions with objects or common interactions with software need to bear a very close resemblance to how those things behave, even if the underlying processing is very different. A couple of metaphors with physical world objects demonstrate this: the trash can icon on a desktop operating system, it is something where you put rubbish in both the physical and software context; and the email icon, as in both the physical and software world you send a mail.
The thing to watch for here is an icon that implies that extra things will happen, or a software process that requires extra action from the user. A contrived example would be if email needed to be synced on the receiving device in order to be displayed. The extra step messes up the metaphor.
Secondly, you need to ensure that the actions performed reflect what the user expects the application to do when they click the icon. As an example I can propose a potential mistake: using a trash can icon within an application which deletes the highlighted content for example. The metaphor itself would be a reasonable one, but the result would not be what the user would expect and that would make the interaction frustrating.
Finally, context is again very relevant, you may find your user group expects different metaphors and behaviour from different actions based on the real world situation which has led to them requiring the app in question. A specialist set of use cases may result in a different group of icons than a general set.
By thinking like this you can minimise the friction for users who are looking at icons without immediately obvious labels and hopefully get some instinctual responses from users and fast learn times.
I would not remove labels altogether though, there are some good suggestions in other comments like to enable and disable labels.