Multimodal interfaces - best practices

Question

What are best practices in using multimodal interface approaches, e.g. combination of haptic interfaces, voice commands, gestures?

Answer 1

Gestures will get more and more important as the touchscreen technology gets popular.

Imagine a popup that closes when user draws a line across it instead of clicking the ok button or the tiniest cross ever in the corner of the window.

The time is coming for the point&drag interfaces to emerge ;) I'm not advocating nor promoting them. Consider it a fact. I still think linux bash console is the unreachable peak in usability :P

Add voice commands to that and we're home.

I'd offer folowing best practices:

• Use only simple voice commands and don't get too big with your vocabulary
• Don't force user to use voice. His wife might be sleeping.
• Look up the natural gestures - like crossing errors with the pen or moving and draging stuff
• Do more tests than design work
• Create tiny tutorial clips and pop them up for the new users, eg. a gif that shows a red line crossing the word to show user how to delete.
• Remember that users might not want to focus all their senses on your application. Dont make them sing and dance and read at the same time.

And last one rather subjective

• If you want a on-screen keyboard wait till you find a company that produces screens that can change the shape of their surface so that users would feel borders of the keys under their fingers. I hate onscreen keyboards so much...

Answer 2

It is getting old (2004), but here's a research article that tries to answer this question : Guidelines for multimodal user interface design.

Reeves, L. M., Lai, J., Larson, J. A., Oviatt, S., Balaji, T. S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J., McTear, M., Raman, T., Stanney, K. M., Su, H., and Wang, Q. Y. 2004. Guidelines for multimodal user interface design. Commun. ACM 47, 1 (Jan. 2004), 57-59. DOI=10.1145/962081.962106

Answer 3

Multimodal interfaces seem to amplify the deviations from unimodal interface standards.

• Using non-traditional or unexpected button designs makes the user pause for a moment to figure out what to do.
  • Extending that to include a voice-interface, if those buttons also have unexpected or uncommon names, the user is left scrambling to remember what to say as well.
• Multi-tiered drop-down menus with anemic pointer-stray zones can be frustrating to use.
  • Extending that to include a gesture-interface, such precise interaction space may make gesture navigation impossible.

As Sruly notes the lack of research to point to, maybe we can come up with a list of common-sense items and do some brainstorming / pruning?

Feel free to edit (made it a community wiki). Note your name after edits or additions, maybe?

Working list

• Make interfaces only as complicated as necessary and as simple as possible. (ML)
• Predictable actions make users more confident. (ML)
• Differentiate commands (vocal, haptic, etc.) as much as possible. (ML)
• Favor combinations of, rather than quantity of, unique commands or actions. (ML)
• When possible, draw from other related applications of that interactive medium (simplified sign language gestures, for example, to correlate with text links or vocal commands) (ML)
• ###

Answer 4

There is very little consumer data on these types of interfaces as they are not very common. As more come out I am sure we will have lots more info.

There will probably be good feedback when the Xbox Kinect launches later this year.

Answer 5

I prefer haptic interfaces with gestures. Its more natural and efficient. It is very tedious and painful to precisely move/close/interact with modals. BumpTop comes to mind when thinking about UI in this manor.