I've used Material Design quite a bit and have reasoned the following drawbacks:
1. Animations are cute but can be cognitively distracting
Material Design's extensive use of animated transitions really helps bring the interface to life and give it a vibrant personality, but animations can ultimately be counterproductive because the human brain is very sensitive to visual stimulation and an animation causes cognitive interruptions which can interrupt a user's train of thought.
Google's principle of "Delightful Details", in particular, creates animations when they are often not necessary for function and can therefore become annoying when repeated, or distracting to thought.
For example, if a user is holding a complex workflow in his head, the following unnecessary animation halfway through the workflow may be enough to break the train of thought:
2. It can be hard to discern interactive elements
Material design doesn't suffer as much as flat design here because buttons are often drop-shadowed or capitalized, making them easier to perceive. But Material Design still uses flat/flush elements so it can be difficult to understand them as clickable. Even the design specification itself has examples of flat clickable text which can be difficult to perceive unless the context is carefully designed. For example, here's a screenshot from the Material Design specification itself:
3. Performance overhead
Material Design makes extensive use of animations, which can carry quite a lot of overhead. For us tech geeks that's usually not a problem because we tend to carry fast phones/laptops, but that is not the case for the rest of the world. Drop shadow, color fill, transform/translate transitions are not dependably hardware-accelerated so the animations can be jerky and unpleasant for users who don't have good client equipment:
Notably, Facebook had to abandon a UI refresh a few years ago because they realized that not everyone has the same nice computing equipment as Facebook does in its Silicon Valley offices.
At worst, the use of taxing animation furthers the digital divide as it amplifies rather than equalizes the difference in client performance.
https://en.wikipedia.org/wiki/Digital_divide
4. Overuse of images and colors can be distracting
Material Design still promotes use of vibrant colors and images in its spec. While this can make an interface lively, this style is prone to overuse and can be very distracting to users trying to get something done. For example, the following menu appears several times in the Material Design documents. It uses an unnecessarily high-contrast header background with multiple layers and off-grid lines which may be fun to look at once, but is functionally crappy because it makes the text hard to read and can get really annoying when the menu is opened multiple times:
5. Affiliation with Google ecosystem
For better or worse, Material Design is a Google-promoted framework and Android is a prominent early adopter, so the framework will probably never be dissociated from a perceptual relationship with Google. That is something to consider for developers looking to create platform independent UX (e.g. an app which also must run on IOS).
One more thing...
My intent here is not to criticize the framework unnecessarily...on the contrary, I really like and continue to use Material Design. The intent here is to document drawbacks because pitfalls and antipatterns are important to understand in any design framework.