As Benny says, it depends on the overall navigation metaphor/paradigm of the app. In one app I built, the metaphor was a home page, and the user opened additional windows on top to view details on entities/objects. They were all full screen (not like pop up dialogs), but the metaphor was "layers" instead of navigation steps. In that case, the X was appropriate to remove the top layer and return to the previous screen. In another app, the navigation metaphor was more of a breadcrumb, where back and forward arrows (like a browser) as more appropriate.
So, if your navigation is stack or layer based, I say X
If your navigation is path or graph based, I say Arrows
I think that is why, since there's a layer or z-dimension to Material Design, the app went with the X icon.