This decision should be a function of the platforms you envisage your users using.
In general, there are a few factors that should definitely be considered (and here I'm limiting my comments to iOS devices, but they apply even more dramatically to Android devices):
iOS devices vary dramatically in available memory. In terms of just devices that support iOS 6, the iPhone 3GS and 4th generation iPod touch has 256 MB vs 1GB in the iPhone 5 and 3rd or 4th generation iPads. For that reason, you should test your application's responsiveness and other usability factors on at least three models (one 256 MB model, one 512 MB model, one 1GB model).
Non-retina iPhones (e.g. the iPhone 3GS) are very different quality screens than more modern, retina displays (e.g. the iPhone 4, 4S and 5). Likewise, the change in physical dimensions for the iPhone 5 can represent a dramatic change in some apps (especially ones intended to be used one-handed).
The issue is even more pronounced between iPhones and iPads, where elements naturally need to be repositioned, resized and realigned to look right on the larger form factor. And again, you need to check the behaviour on a Retina and non-Retina iPad, and you should probably check an iPad mini too.
Depending on your app, you'll probably need to be able to test the behaviour in three states: a strong Internet connection, a dodgy (high latency, low speed) mobile connection and no Internet connection.
Of course, it's not always feasible to user test your app in every scenario (although your developers really should be testing on all target hardware since the iPhone simulator is not a reliable test). The only way to narrow down the list is to start doing some research into what devices you expect your users to use. If your app is a mobile navigation solution for drivers, you should probably prioritise phones over iPads, and it's reasonable to test on devices with functional GPS units. If you're building a bleeding edge game with incredible graphics, you probably aren't going to be targeting the iPhone 3GS anyway, so prioritise your testing on the more indicative devices.
If you have a previous version of your software, you can view your download logs and prioritise testing based on the actual devices your users use.