MS Windows 7 UX Guidelines(1) has standards and guidance for mouse use for desktop apps.
Dragging with the left mouse button down (with no modifier keys) is the standard for selecting multiple objects withing the rectangle delineate by the drag. User may find it confusing if the pans instead, although showing a “hand” instead of an arrow for the pointer could help that.
In the standards, shift- and ctrl-left-clicking are used provide alternative ways of selecting multiple objects, so, if you follow the standards (as commenter bloodyKnuckles mentions), they are not available for zooming or anything else. For example, shift-left-clicking will select all objects in a rectangle delineated by the currently select object and the object you’re selecting with shift-click. In a crowded pane, this means a that shift-left-click dragging can select a bunch of items and start to move then. In any case, users are not going to anticipate the behavior you are considering, and they may get confused or commit errors when going from one program to another
The Windows UX Guidelines recommends rolling the mouse wheel for vertical and tilting the mouse wheel for horizontal. Using the mouse wheel for vertical panning is very common, but, personally, I've never seen tilting the mouse wheel do anything, and I don’t think my Dell mouse even supports tilting.
As an alternative to tilting, you may want to try using the mouse wheel with the shift key. No, I don’t know which way would be left or right, and, yes, Windows UX Guidelines say this is disorienting and unexpected. However, it is consistent with general recommendations regarding the shift key and at least keeps horizontal and vertical control mentally proximal to each other.
A better alternative is to press the mouse wheel (like a button) and drag the mouse to pan, much like you’re thinking with the left mouse button. This is used in browsers like Chrome and Edge, and my CAD program, so it appears to be an existing, if not widely known, convention.
The Windows UX Guidelines recommends zooming when the user rotates the mouse wheel while the Ctrl key is pressed(2). This is used in Microsoft Office, browsers Like Edge and Chrome, and my CAD program, so it seems to be pretty common.
You can have it zoom in on the point when the pointer is position, so the user can zoom to a particular framing, rather than the user dragging to set the zoom frame, and thus you don’t necessary need drag-to-zoom. This also allows the user to zoom out easily, which your design doesn’t support.
Whether you use your own interactions or the standard ones, use of modifier keys is not particularly discoverable. Even if you follow the standards, user won’t necessarily anticipate that those interactions are in fact available in your app, so they may end up thinking there is no zooming or panning.
Thus, you may want to consider a more visible indication of the interactions in addition to the “shortcuts’ of using modifier keys and the mouse wheel. This can be done by providing a “palette” (toolbar) of “tools” for the user to select and change the behavior of the mouse. You can have separate tools for select (default), zoom, and pan, each changing the pointer to indicate what is in effect at that time. For convenience, you may want to assign shortcut keys (shown in the tool label or on hovering) to each tool so users can learn to select them from the keyboard instead of slewing back and forth across the screen. That could be as fast and easy to use as using modifiers keys on the mouse buttons/wheel.
(1) There’s also Windows 11 design principles that cover mouse interactions, but it doesn’t seem to be as complete.
(2) The Windows UX Guidelines also suggest zooming with an unmodified mouse-wheel-turning for windows and panes “without scrollbars,” but I’d ignore that. It’s an arbitrary distinction, and, with programs that do zoom with the unmodified mouse wheel (Corel, I’m looking at you), I routinely zoom when I mean to pan vertically. Very annoying.
(3) My second favorite mixed metaphor in GUIs, right below wallpaper on your desktop.