As noted in the comments, you are limited in what you can practically implement because you are fighting to fit an image of x dimensions into a viewport of y dimensions.
As I see it, your options are:
Load images into the viewport, centre vertically and horizontally, if one of their dimensions is less than the equivalent for the viewport, zoom the image to stretch. Also then provide panning(/zooming) functionality, such as a thumbnail the user can move over to pan the main image (think the same as a lot of clothing outlets)
Orient the viewport appropriately for the image, vertical for vertical images, horizontal for horizontal ones etc.
Personally, if you want consistent viewport dimensions, I'd go for 1 - providing a thumbnail so users can see the whole image, then allocating the rest of the space for a zoomed (or 100%) pannable version.