It depends on how you approach this component. Every component's identity, operation, and state should be available to assistive technology to make it accessible.
When an arrow is on keyboard focus, screen reader should inform the user about the identity of the element (Up arrow) its state (clicked/ unclicked) and operation (press enter to increase the ...
My recommendation for the last couple of years has been to layout forms
Issues like the one you are describing is a good reason for following this layout, but so is the small screen use case where they visitor may not have scrolled down far enough to see the error message or note.