You can add an overlay on the graph if the data and filters mismatch prompting user to refresh / reload / load the graph again.
However it does not end there. I wonder if users are expecting the graph to refresh when they select the second value. How your application behaves in other similar scenarios? Are actions auto triggered on selection or there are designated buttons to that? what is the consistent behavior across application? Any particular reason to call it refresh? These questions need answers, for user's mental model expects certain things a certain way.
If the refresh flow can be invoked as many times as user wants to get latest data, I think it should not be closely linked with the date selection. You can always have a separate panel like configure chart/graph which has graph configuration operations you are offering. This separate panel near the chart, can have a submit button to render data when user has finished editing. Refresh can still be there independent of chart configuration.
something like this.
This is out of the first impression of your use case. I might be wrong, or may need to change it significantly, but the point I am trying to convey is keeping your simple refresh flow different from chart configurations. Chances are users will refresh more often than they change the configuration of the chart.
I hope I have been somewhat helpful.