If a hacker is trying to hack, should I present a pretty error page or redirect somewhere instead of a generic/cryptic error? I feel that the hacker knows what they are doing, I know what they are doing, they aren't supposed to be doing it, so I have no real incentive to put the effort into their experience. It was a hacker because there was a high load of attempting to crack passwords at /users/login.
There are couple of user bases to consider here.
The clueless newbie who is not sure what goes into a form or who just migrated from an old system where he used to enter search queries to retrieve data and does not know that now he can just retrieve content by using the filters provided. An user who does know what he is doing but is not aware of the negative connotations will also fall into the broader version of this group.
While I can understand your objective to be aggressive towards towards the first group, I don't think your second group should suffer the consequences of a careless mistake or lack of knowledge which led them to make this mistake.
To take a simple analogy, take the case of a person who deletes the system files in Windows by accident opposed to someone who just formats the system files intentionally. Both of them get the same error message
This doesnt convey anything to the first user who did it by accident and by providing him with a cryptic error page you are just going confuse him further.
Check out how Google handles a suspected virus activity with a firm but helpful message
Similarly here is another example of how google handles a server which was accidently sending automated results to Google (do note the user was not aware of it, which puts the person in the user group of not being aware that he was making a mistake).
I don't usually run Google searches on FireFox only to see some detailed information as provided by SEOQuake. Yet, today I was searching for some potential answer to a problem I'm running into with MS-Access which crashes a database and ran those on FireFox instead of my favorite: SeaMonkey.
As I was testing, all of a sudden I got the screenshot as shown in the figure below. Although it did not feel like I was sending automatic requests to Google, the SEOQuake toolbar does so on all the results! That means each time I do a search "I" send at least 11 requests to Google. After 3 or 4 searches with pretty much the same terms each time, that's well over 40 requests within 2 or 3 seconds.
That's the first time I got this screen, I'll have to think of turning off the SEOQuake bar whenever I do intensive searches like these.
alert(). It is the bad UX he deserves! (Alternatively, redirect him to a song by Rick Astley.)
If there is a chance it was a valid user making a mistake, then you should provide a message that will help put him back on the right track. Something like "Sorry we do not accept strings containing HTML markup".
A classic way to deter a hacker is to simply delay his response. (You can find many systems do this when you enter an incorrect password, rendering bruteforce attacks impractical.) If you make him experience increasingly long delays before he receives each failure response, he will either give up or timeout. A valid user will tolerate some delay while they strain to remember their password.
Hackers don't deserve good UX, they deserve bad UX or no UX. So don't waste your time on this unless it could improve security which I can't see how.