Maya’s “hotbox” menu system has apparently been subjected to fairly extensive evaluation, so you can use its characteristics as a starting point for testing of your app. You can read more on the designers’ rationale and lessons learned in developing hot box in:
Kurtenbach G, Fitzmaurice GW, Owen RN,
& Baudel T (1999). The Hotbox: Efficient access to a large number of menu-items.
CHI 1999, May, 15-20.
Note, however, that what they describe looks different than the "Figure 33" screenshot you're showing.
Dismissing the Menu
The “safest” approach is to follow the conventions of a context menu (what usually appears with a right-click) and remove the menu when the user clicks either on a command in the menu or outside the menu, the latter being the means to cancel selecting a command. This should be consistent with most users’ expectations and will be thus be discoverable.
The chief alternative is to use a “quasi-mode,” where the menu remains visible only as long as the user is actively sustaining an action. In Maya, this is action is holding down the space bar. This makes dismissing the menu naturally easy and discoverable –if you don’t want the menu, stop doing the menu-action.
You may be able to do this with right click (show the menu as long as the user holds down the right mouse button), but there may be a few issues:
It’s not the way right-click usually works, so users may have trouble getting the hang of it. Once they do, this habit may interfere with using right click in other apps, where pressing and holding down the right mouse button typically does nothing.
This is awkward if you have any pulldown menus. Users can pretty easily hold down a mouse button and move the mouse as long as they move in a straight line to a target then release. It gets a lot harder if they have to change direction, like when navigating a menu cascade. This is one of the reasons that Mac apps, which have similar behavior in for their menu bars, have resisted cascade menus.
It may be technically difficult for the app to react to a mouse button press. Context menus generally appear on mouse button release. I’m pretty sure there is a mouse-press event you can use, but there may be some technical issues with it.
Instead of right click, you could use a key like Maya does. This gets around all of the above issues. A user can easily hold down a key with one hand while slewing the mouse any which way with the other. It also allows the user to select multiple commands in sequence without needing to re-display the menu each time. That's apparently a major advantage for Maya, and may be good for your app too.
It’s understandable that you can’t use the space bar like Maya does. If you have no menu bar, how about using the Alt key? Normally the Alt key provides keyboard access to the menubar, so it may already be tied to “menu usage” in some of your user’s minds. Nonetheless, any use of a key will have discoverability issues, and you need to make it clear to new users. On the other hand, once they learn it, I bet they’ll never forget it.
Number of Items
Hot box was designed to accommodate a large number of menu items. Maya apparently shows about 30 menu items (of which most apparently open cascade menus for more items). This is consistent with research showing that you can have such large numbers of menu items on a single menu as long as you visually group the menu items into related clusters, as Maya does.
Your main concern is not so much the number of menus as the size of the area covered by the menus. You can reduce the area while still having a large number of menu items if you make the menu items small, but this makes then harder to read (due to small font or the use of icons*) and slower to click. Transparent menus are likely necessary for a large number of commands (e.g., over 12), but even with transparency, you’re going to somewhat obscure the background the user may want to see. The optimal menu size depends on what users need to see behind the menu.
Optimal Menu Item Placement
You need to balance two issues:
Put more frequently used commands near the pointer for faster access.
Group commands into meaningful groups, putting similar items next to each other to aid finding and learning the items
Fortunately with a pie menu like you’re proposing, you have substantial flexibility. You have at least four locations close to your pointer (left, right, above, below), so you can have four main groups of commands where each can have its most frequent command right next to the cursor. Can’t do that with a context menu.
There are also a couple things you can do to control speed of access independently of grouping.
If you have a frequently used command that nonetheless needs to be far from the pointer in order to be with right group, you can make that item larger. Large menu items are not only easier to see, but easier to click –click speed is proportional the logarithm of the distance to an object divided by its size (Fitt’s law).
If you have an infrequently used command that nonetheless needs to be close to the pointer for grouping purposes, you can put it under a “split button” for the most closely related command. This leaves space open for a more worthy item.
If I had to guess, I’d say Western user will look first to the upper left from the pointer for menu items, so you also want your relatively more frequently use items positioned to left and top of your pointer.
Even more important is to be consistent across contexts: keep the same commands in the same place. One of the advantages of pie menus is that experienced users development muscle-memory for the commands, and execute them with a quick slew of the mouse without really looking carefully at the items themselves, in essence making a gesture-driven UI. Consistent command location encourages this.
*Maya designers apparently had more success with text labels than icons for their commands, even though Maya is a graphic creation application where one might expect icons to do relatively well.
