Root Cause: Implementation Driving UI
The whole idea of a single consolidated Options dialog seems to be a consequence of programs having configuration files. The Options function would read the configuration file, present it to the user in a dialog for editing, then save the changes back the configuration file. Perhaps some other programs do the same thing, only reading and writing to the Registry. The implementation, rather than the user’s tasks, is driving the UI design, which often a bad thing. It makes it convenient for the programmer, but not the user.
With the possible exception of when a user first installs a program, a user generally doesn’t think, “Today I’ll change my options.” What happens is the program is doing something specific that isn’t what the user wants (e.g., by default pasting to match source format). So users (if they’re savvy enough) think, “Maybe there’s an option for it,” and embark on a major safari to hunt it down in a jungle of options, assuming it exists. No wonder it doesn’t work.
Addressing Discoverability: Options Proximal to Affected Features
The solution is to put the options with the feature it affects. Don’t have all options in one massive dialog. Distribute them around your menus and dialogs. For example, for changing the default pasting, the Edit menu has a Paste Options menu item below the Paste menu item. Paste Options opens a simple dialog to select the default pasting behavior. Or there’s a Paste Special dialog that includes “Make this Default” so when users select special pasting behavior for a particular instance, they can also make it permanent. This improves discoverability. Or maybe there is simply "Editing Options" if there are also options for Copy, Undo, and Delete (otherwise, your menu has nearly as many "options" menu items as all the other more important ones). When the user executes the command in its menu or ribbon tab, they see there are options for it. It reduces workload, because the users don’t have to search through a zillion irrelevant and questionably organized groups of options to find the one they need. Options are organized exactly like (with) the associated features, which the user already knows or else couldn’t be using the features.
Options by definition are preserved between sessions, so they are most appropriate for things that are very rarely changed. For this reason, it’s important not to clutter your UI with Options menu items and command buttons since they’ll get in the way of what users are usually looking for. You probably should have no more than one Options button per dialog and one Options menu item per menu or ribbon tab. That may not be as strict as it sounds. Going through MS Word’s options in your example (which is among the more extensive), I see there would be:
A Cut/Copy/Paste Options menu item on the Home tab.
A Format Marks Options menu item on the View tab.
A Proofing Options menu item on the Review tab.
That’s about it.
The rest of the options are either put under an Options button in a dialog/page (like Save and Print options), or they are options that aren’t associated with a menu or dialog, so they have to be handled differently (see next paragraph).
Addressing Inconsistent Locations: Have a Preferences Menu
There will be some options that don’t have an associated menu or dialog box, such as “select entire word” in your example. So you still need some kind of centralized place for setting them. However, they are fewer than the total number of options, so it makes them more findable, assuming it isn't best to have the same options both proximal to their associated feature and in a centralized location to maximize their discoverability.
You are correct that there is now no standardize place for them. Microsoft had “Tools” as a standard for a while, but “Tools” is like “Advanced” –a junk drawer of things that don’t fit anywhere else. Apple’s standard of putting Options in the application menu (the menu with the name of the program) sort of makes sense given the paragraph above because these are options of the entire program, not just a specific feature.
However, my advice is to have a Preferences menu at the top level (a pulldown menu or ribbon tab). The place for options is their own place, not under any other menu or tab. This maximizes discoverability: the users see “Preferences” (or “Options,” or maybe “Customize,” depending on your users) every time they open the app. It doesn’t matter if the user is expecting Options to be under Tools or Edit or the application menu. They immediately see it at the top, and don’t bother to look anywhere else.
Space or clutter is generally not an issue especially for menu bars. After File, Edit, View, and Help, most apps don’t have a whole lot else. If there are only a few options (eight or less), the Preferences menu has toggling menu items to set them directly –there is no dialog box. If there are a lot of options, split them up into separate dialogs and access each with a separate menu item (rather than have separate tabs within a dialog). For in-between numbers of options, have a mix of toggling menu items and a dialog or two.
Addressing Poor Explanations: Active Help
As addition to, or perhaps as a better alternative for, a Preference menu, give Help the ability to change preferences. Take “smart paragraph selection” in your example (whatever the hell that is). A user is not going to think, “This dang smart paragraph selection! I wish I could turn it off!” They’re thinking, “This selection is doing something weird; why is that?” So they naturally go to Help to learn more. If the index and search are decent, and the content clearly written, they find out about this smart paragraph selection, and how it works. Maybe once they understand it, they want to keep it. However, if they don’t, then Help should provide the means to turn it off right there in Help. It doesn’t direct the user through multiple steps in an Options dialog to effect this. It’s not like users need to learn that –they’re only going to turn it off once then forget about it. Let them do it right there.
Putting options in Help means searches don’t have to match the options name exactly –it can match any part of the explanation. They can also match any synonyms, assuming your search provided for that. Users get the whole explanation, so they can better determine if it’s the option they want or not.
And Installation?
Consider on installation presenting the user with a list of commonly change options, showing what is default, and allowing the user to alter them right there (e.g., default folders, devices, and such). However, often users need to use a program before they discover what does or doesn't work for them. Thus, it's important to make options reasonably discoverable after installation too.