With approach 2 (block invalid characters), the user might type something quickly and end up with a result which looks nothing like what they wanted.
Let's say a user-name field should only contain standard unaccented characters. A user could enter
Juergen without a problem; but attempting to use ü instead of
ue would be ignored and the user would end up with
Attempting to create a username like
$4nta Cl4u$ would be similarly stymied:
This is bad: the user would have to work out what had been omitted and why and then correct it. Approach 2 should be avoided.
Method 1 (validation message) is therefore to be preferred — and this should be inline during entry rather than returned after the form is submitted.
I would prefer adding the user's invalid characters to the message as they are entered. Thus the message
You cannot use ü would appear as soon as that character was typed; and in the second example it would end up as
You cannot use $4[space]. (Actually using
[space], not a blank character.)
In Method 1, it would be useful to make some help available as well in order that the user can find out exactly what is valid. That could appear as a tooltip, or open a separate page.
download bmml source – Wireframes created with Balsamiq Mockups