Several ideas come to mind.
No tabs in rich text box.
The simplest solution is your suggestion to simply not support typing tab characters in the rich text box. I expect users will naturally enter 4 to 5 spaces if they discover that the tab key doesn’t work for them. You may want to even consider converting any series of four to five spaces into a tab character, sort like how MS Word “auto corrects” (c) to be the copyright symbol. That’s probably good enough if your users rarely need to enter a tab character, and there’s no other good reason for users to enter a series of spaces.
Ctrl-tab
The closest to a standard is to use Ctrl-tab in place of Tab to shift focus out of the text box. Ctrl-tab has been used to shift among interior windows in an MDI, for example, so it’s somewhat of an analogy. The rationale is the “ctrl” means “make bigger” so if tab only shifts the cursor/focus a little (inside the rich text box), then ctrl-tab should shift the cursor/focus a lot (out of the text box).
The problem is you’re already using Ctrl-tab to shift among your toolbar controls. However, it sounds like you don’t want a toolbar but a tab control –something to show different sets of controls inside the same region. If you use a tab control instead of toolbar (or make your toolbar act more like a tab control), then the bank of tools/tabs becomes just another control to tab to (with the tab key). Once focus is on the tab control, the user uses arrow keys to move among them. Ctrl-tab is now free to use to exit your rich text boxes.
(Actually, for better consistency with other apps, you may want to use a tab control instead of a toolbar regardless of how you solve the tab key problem.)
So that’s the closest to a standard. However, it has poor discoverability, so it may not be suitable for your users given they rarely use the app. Internal consistency is better if the tab key always moved among controls. Certainly before committing to using Ctrl-Tab (for anything), test it on some users to see if they naturally try it unprompted.
Editing Window
Do users enter a lot of rich text, like, the equivalent of an 8.5x11 page? Would showing all the rich text at once take more space than you need for the other sets of controls? If so, maybe the rich text box should be in a window by itself. On the main window, show only the first portion of the text in a read-only textbox, with a button for Editing that opens a full-size resizable window with all the menus and toolbars for editing and formatting the text.
Normally, edit-in-place is preferred, but a dedicated window can be better because the user can see more of the text at once, and it may be easier to understand and access the editing/formatting controls (e.g., you can have Cut, Copy, and Paste as proper menu items, rather than relying on expertise in short cut keys) It also opens up many possibilities to better support the users’ tasks. The user can Save as they go. Maybe the user can Save As and Open local versions of the rich text so they can manage a library of alternates to apply in different situations. They can print out the text to see it as the customer sees it.
As a side benefit, a separate rich-text editing window solves the tab problem. Users don’t tab out of the rich text. They close it.
Alternative to tabbing, not tabbing
Finally, maybe you should find an alternative to inserting a tab character in the text box, rather than an alternative to tabbing out of the text box. Among your rich text controls (e.g., Bold, Italic), you can have a Tab button, or an Indent button that makes tab-like indents for the user. You can give it an accelerator like Ctrl-T. Now the tab key can take the user out of the rich text box. The main advantage of this is that having a visible button provides discoverability the first and second options don’t have.
