According to Google's navigation guidelines, the correct behaviour for your described case of hitting back button is to return to the previous screen, which will be the previous category.
They outline 3 cases for the back button.
When details screens are independent and only accessible via the Hub screen

This is your above case where your categories are independent within the app. Because the categories are considered to be distinct items, the back button returns you to the previous screen.
When sibling screens are accessible using swipe action

Because the sibling screens are easily accessible by swiping, they are considered to be cards on a single screen. Hitting the back button returns you to the main element.
When app is launched via widget

If the app is launched from a widget that takes you directly to a "details" page. The action of the back button should first take you to the main screen of your app. Only if the user taps on back again, is the app closed. The reason being when you tapped from a widget into the details. It make sense to allow the user to zoom out to look review the context which is provided by the main app screen. That is why it does not directly close the app.
They provide more examples on app to app navigation behaviour in the link. If you're interested in the subject.