We're working on an internal business app, and throughout the process the users have been very involved in all aspects of the design. They have explained their workflow, terminology, etc. and the app models that as closely as possible, with their input at every step.
Most actions in the app can be undone, but there's one action that (per the users' instructions) can't be undone without going into the database and manually changing data. In order to confirm this action, the user has to click "yes" on a modal alert. Alerts are used very rarely in the app; there's only one other action that will bring about an alert. Neither action is performed very frequently.
However, the users are still somehow occasionally clicking on the wrong action and confirming their choice in the alert (presumably ignoring the text on it). What's the best way to remedy this?
The text in the alert is a bit long, so we're looking at making it shorter and more to the point, and maybe renaming the buttons so they aren't just yes/no, but we don't know if that's enough. Is there some other way to draw the users' attention and make them really realize what they're doing? Are there proven ways to reduce errors like this?
ETA our solution: For now, we're focusing on changing the text and appearance of the alert dialog, as per Matt Lavoie's and André's answers. We made the text shorter and more to the point, added a caution icon and a clear statement that the action can't be undone, and changed the buttons to say what they'll do and not just yes/no.
We also changed the text on the button that launches the action to be red and in caps (all the other button texts are in black and just first letter capitalized).
Finally we made the entire background of the app go bright red when the alert dialog comes up. It's quite jarring, so the users should definitely notice it.
We mentioned the changes we're making to the users, and they were totally in favor of it. If these aren't sufficient we'll add additional input requirements to the alert, as in Andrew Leach's and JeffH's answers.