The first issue is, which x does what?
You have two "x" icons which may do the same thing, but perhaps not. If they do the same thing - why do you have two? If they don't do the same thing - you're using the same icon to do two different things!
My concern is that a user may hit the "X" when they intended to hit the arrow, accidentally clearing their selection.
Yup. That's likely going to happen.
The notion that a dropdown box has revert to the default value is not a standard paradigm. The default value should be listed in the dropdown, so the user can also re-select if necessary.
Although it is common, having a "Select..." option (or similar) is not preferred. Your default value should be a real value. If the value is indeed a real value it should be placed in its proper place in the list order, not at the top for the sake of being "the default".
Maintaing a standard dropdown would be your best approach. Remove the "x" and simply select the default value, placing that value in the last as it should be based on the sorting order.