Boolean logic is inherently difficult, especially when it gets complex (up to five levels nested!). It may not be possible to come up with a single representation that is both easy to understand and compact enough to provide an overview. You may need to have two representations.
Space-consuming but Easier-to-understand
The OSX solution isn’t a bad one. It’s probably a good idea to substitute “All” and “Any” for “And” and “Or” since the latter are ambiguous in vernacular usage. It’s also probably easier to see and track nested conditions/criteria with the indented hierarchy rather than with pairs of parentheses.
The OSX is a pretty text-intensive representation, even with the hierarchy. Perhaps your users would do a better with a graphic representation employing a plumbing or electrical metaphor. Such representations may do a better job of highlighting any redundancies and contradictions in the user’s thoughts to help them refine the criteria to be something useful. See my response to Intuitive interface for Composing Boolean Logic? for research and prototypes of this approach.
Query-by-example (QBE) has had a fair amount of success. For this, users use the same forms they use normally but fill them out with field values that represent the kinds of results they want to include. It handles ORs and ANDs especially well, but handling ranges (e.g., Greater Than) can be kludgey. Plus, using QBE to specify nested criteria can be very time consuming.
In any case, the UI for saving and retrieving such representations/filters can be much like what you'd used for saving and retrieving a file.
The above options all consume a lot of real estate, so it doesn’t solve your problem of providing an overview (e.g., in a list of saved filters). For that purpose, it’s hard to beat a string with nested parentheses for its compactness. That may not be such a bad thing for your users since they’re experienced with Excel –if they write their own formulas, they may be used to it. You may even be able to use Excel-like symbols (e.g., =, >, <) rather than words for a more compact representation. Furthermore, you can exploit the users’ familiarity Excel’s color-coding of matching parentheses in order to better visualize the nesting. I’d explore redundant coding, such as use of underscores. Maybe something like:
Cooper and Reimann’s About Face includes an example for making such a string easily editable for adjusting the criteria (at least the second edition does; haven’t looked at the current edition). However, for complex logic, it would be better to use the string to provide the overview and when the user clicks on an appropriate control, the string expands into a full representation like listed above.
You could try to automatically simplify the string and the full representation by “optimizing” it with logical operations and removing redundancy by following a set of conversions. For example, the criteria in your OSX example could appear as:
The disadvantage of this is that the user may no longer recognize representation as equivalent to their own concept of the criteria. Not everyone is up on their symbolic logic. Did I mention this is an inherently difficult task? You’d want to be sure to test it.