Suppose the user receives a popup (modal) message for successful task, then clicks the back button on the browser.
Should the modal popup message appear again even though the user hasn't performed any new task?
It should simply go back to the previous state (i.e page) and not repeat the previous action. Otherwise it would seem that your app/website has a bug, since this is not the expected behavior in most systems or web applications.
Please refer to the workflow below for reference.
Well, "should" is a strong word.
Right now, the predominant pattern is that the back button never does this. Based on this we can understand that any user is likely to be surprised by the behavior of re-showing a modal dialog when clicking Back.
Whether this is a preferable behavior... it's surely a matter for a demographic study. One might expect that trained users would be confused or unsatisfied by the behavior, since they are trained otherwise. Users new to web browsers may come the be trained by this pattern, and prefer it.
I think the latter is very much predicated based on whether a user is both sophisticated enough in attending to the Back button, while also being cognizant enough of the workflow of the web app in question.
My personal opinion is that naive users will think of their app workflow 'spatially' in a way where previous interaction-centers (each major page) are the landmarks in a timeline. The consideration of that timeline is predicated on awareness of the 'back' concept and button.
Given this, I don't expect most users will think of modal dialogs as major interaction centers, and such points will blur in comparison to memory of other major areas. Thus I think that naive users will not expect the back button to included modals in the workflow timeline.