Should I limit inputable characters in an internationalized multiplatform app?

Question

Suppose I am releasing a mobile app and website and make it available to English-speaking and non-English speaking users.

We already support UTF8 encoding throughout our technical stack. However, is allowing user input of the full range of UTF8-expressable characters generally a good policy? And if not, how do people

In particular:

• usernames
• passwords
• user generated comments

may be provided as input characters on one device (via a touchscreen keyboard) and there's a need to render of re-input the same characters on another device.

For example,

• A forum post. Here is the Skull and Crossbones character "☠" (UTF 9760). Some third-party Android keyboards provide an easy way to input it, but you may find it renders as a square box in your browser. Generally we may wish to use different fonts on different devices (e.g. different fonts for high ppi devices vs. low ppi devices) and so we run the risk that the user can freely input text on one device (and see it correctly rendered), and that same character is not renderable on the another device in the alternate font (= UX fail). So, it seems sensible to prevent the input of such characters through form validation. If so, how is the set of acceptable characters usually decided? (For example, is there a big list of which characters are available in common web-safe fonts).

• In the case of a creating a new password, the Android keyboard invites people to enter the ¥ (yen) character. However, if we allowed them to do this, most US users would be stuck when they come to login to the same service on a PC who's keyboard has no obvious means of entering ¥. Oops - UX fail.

I've lived my life in ASCII. What are the best practices here?

Answer 1

You should never limit it, unless you have a very good reason. The users can sometimes innovate in unsuspected ways (it is said that tea bags where for test, and the consumers began to ask it in that way).

• Comments: A big NO, unless your system has some very strange use. You could want to talk about a Greek person (say Socrates: Σωκράτης) in Spanish (Sócrates). Or you could want to write a formula: ∃n∈ℕ P(n,n,25)
  • Later, you could apply some restriction if something turns annoying (users abuse some Unicode functionality).

• Password: The only downside I could imagine is the user being unable to repeat the password, in this case they could reset the password by mail.

• Or you could provide a on-screen keyboard. This have some advantages, like:

  • You could change the order of keys by random, and avoid some keyboard and mouse loggers.

• User names: Here you can have good reasons (users imitating other's names). So you can implement several kind of restrictions:

  • Only latin: Bad for users, easy to do. Good for compatibility (to use the user name as email address, as Google and GMail).
  • Only one language: You should have a list of every permited chars for every supported language.
  • Avoid collision by similarity: This is better for users because you could also implement sound similarities (ralf vs. ralph), visual similarities (RALPH vs. RAL9H), and char reordering (ralf vs. rlaf). But it is a complex task.

Answer 2

I don’t think you understand the purpose of UTF-8 very well. It is just an encoding that allows you to store all kinds of characters. Choosing the encoding is more a technical decision and since you are creating a multi-lingual application and have the ability to choose UTF-8 it looks like the encoding you need.

It is your application that decides what to show to or accept from your users, not a database or file system related encoding. It is not easy to change a file system, but an application's behavior may change regularly, so UTF-8 may be the best choice here to avoid any limitations or difficulties later on.

Use application level validation and filters to control the data between user and application.