The mental image of time is indeed thought to be influenced by language and culture.
Scientists discovered years ago that spatial representations of time are affected greatly by linguistic conventions. If English is your native tongue, you're likely to think of time as moving from left to right, but if Arabic is your language of choice, time moves from right to left. These differences can be traced to the direction in which one's language is written, either left-to-right or right-to-left.
From the Psychology Today article Is Before to the Left of After?
Interestingly, some cultures don't follow the left-to-right or right-to-left convention and instead associate time with the compass:
The languages of the Pormpuraaw, a remote aboriginal community in Australia, include relative spatial terms like left and right, but they're rarely used. Instead, speakers of these languages rely on absolute terms that correspond to the four cardinal directions. Pormpuraawans generally think of time as moving from east to west, just as the sun does in its daily journey.
With regard to charts, time is generally the independent variable and it is convention for the independent variable to be presented on the x (horizontal) axis.