The semiologic point of view:
An x mark (also known as a cross, x, ex, exmark or into mark1) is a
mark (x, ×, X, ✕, ☓, ✖, ✗, ✘, etc.) used to indicate the concept of
negation (for example "no, this has not been verified" or "no, I don't
agree") as well as affirmation (for example in election ballot papers
or in x marks the spot). It is often used opposite the check mark or
tick (or the O mark used in Japan, Korea and Taiwan).
The UI history point of view (based on semiotics as well) dates it to Atari TOS 1.0, and later it resurfaced in Windows 95:
[x] is a true icon, not representing a letter but representing an
action, and only adopted to represent ‘close’ well after the
development of graphics-oriented operating systems. The first
appearance of [x] in GUI design was likely the Atari TOS, possibly
influenced by the Japanese batsu and maru conventions. Thanks to a
last minute design change in Windows 95, and the mass adoption of
Windows worldwide, [x] has become the standard symbol for ‘close’, a
symbol that dominates web, app and software design today.
I've also heard the version of x representing crossed arms as in "defense" or "don't want", but really couldn't find anything to support this claim, so take it with a pinch of salt, just mentioning because always found this curious and maybe someone can shed a light on this
EDIT: re-reading the link I posted, the "crossed arms" is elliptically mentioned in the article. The concept of batsu is actually 2 arms crossed:
Batsu. In Japanese culture, the batsu (literally: ×-mark) is a gesture
made by crossing one's arms in the shape of an "X" in front of them in
order to indicate that something is "wrong" or "no good".
so guess the version is correct