Well, I am almost positive that there is no such research that resulted in a recommended specific number of characters to error/warning/alert messages. However, you first can investigate other researches that could support the idea of reducing text and amount of reading. NNG research -quoted below- resulted in a law that users generally do not read Computer Documentations and shy away from doing any extra reading that is not essential to a task in hand.
"Finally, you probably already know Nielsen's First Law of Computer Documentation: people don't read it. This finding is even stronger for websites, where users truly shy away from any reading that is not essential to their task. Click on Help? Never."
Second, you can use common sense that is supported by facts. More text means more reading which requires more time! Take Form Design as an example. In order to achieve an increase in the form completion rate, designers struggle to reduce the completion time (Time!).
Have in mind following the guidelines while trying to reduce the text! (Here) NNG Error Messaging Guidelines.
One main guideline tells you to not just point out the error but also tell the user what to do. If we take this example:
e.g. "This name is already taken. Please enter another name".
If I were to reduce the text here while having in mind that I still need to state the error and tell the user what to do next. I will maybe suggest this form:
"Please enter a name that isn't already taken".
Hope that helps!