1

When performing usability tests on mobile devices the results are influenced by the devices tested on. For example different sizes, OS-versions, resolution etc. To conduct a thorough test it is important to make sure a representative set of devices is tested

What sources can I use to decide which devices to buy / use for testing of new (mostly native) applications and how to stay up to date with the available devices.

||improve this question
  • If they are mostly native applications then don't you already know what devices they are going to be run on? – JonW♦ May 31 '13 at 15:59
  • No, an ios device could be an ipad, ipad mini iphone 3, 4 ,5. Same for android, htc desire, htc one, samsung, lg, etc. Different OS versions, different devices, resolutions, skins etc – TomDoes May 31 '13 at 16:02
  • This is especially tricky on Android due to the fragmentation of the platform across devices. – Charles Wesley Jun 6 '13 at 15:48
2

While it's accurate that the user experience of a mobile application is dependent on the device and the version of the operating system that the device is running, I don't think that it's necessary to conduct usability tests on a large number of devices. For valid results, you would need at least 5 participants per device, and I can't imagine having sufficient time or resources to run that kind of usability study. If you actually did have sufficient time or resources to conduct that kind of research, your time would be better spent doing other research that would help you learn more about your application and your users.

You can learn what overall market share is of mobile devices, but this doesn't tell you anything about your users. You should learn what devices your users are using, and use this information to inform your decision about what devices to use for testing purposes. I would choose no more than three devices for usability studies. This will give you sufficient information to make design decisions, and you can watch online reviews and other feedback mechanisms to see if the application works particularly well (or particularly poorly) on a specific device/OS. If this specific device/OS is one that is relatively popular amongst your users, then it's probably worthwhile considering whether you should conduct additional research with that device. If it's not, however, you and your development team should determine whether your resources would be better focused elsewhere.

||improve this answer
  • I agree testing the interface on all devices with 5 participants is not an option. But I also agree with Kit Grose's answer. I need to know how screen size, physical memory etc. affect the app. In most cases I won't need 5 participants to decide if it's ok or not, I could decide if it's good enough or not. But I still need to test on a device. – TomDoes Jun 3 '13 at 6:56
  • @TomDoes: Doing hallway usability testing and functional testing against your full array of hardware may be enough to ascertain the degree of fuller usability testing actually required to be a meaningful sample. – Kit Grose Jun 3 '13 at 23:33
  • This sounds like a point where you can partner with your quality assurance person/team (if you have one) and help them understand how they can help improve the user experience. – nadyne Jun 4 '13 at 0:03
3

This website lists screensizes and resolutions for devices but also gives a quick glimpse of what people are currently using.

http://screensiz.es/

||improve this answer
  • When I open the link, I don't get anything but a header. No content is visible – James Jenkins May 31 '13 at 19:01
  • Really? That's quite the bug. What browser are you using? I can see it fine with Google Chrome. – Adam Waselnuk May 31 '13 at 19:21
  • I am using IE, just tried again and noticed it is loading VERY slow. I have a fast connection, so am not using to a page taking this long to load. Still waiting - Check , yep still not loaded, check again, Still not displaying anything. Ok almost 30% loaded, closing the window – James Jenkins May 31 '13 at 19:25
  • Well that's just terrible. I don't blame you for closing it. My apologies for suggesting it, but I have personally never had an issue loading that website. – Adam Waselnuk May 31 '13 at 19:27
  • Seems to working fine now. Looks like a good resource. – James Jenkins Jun 1 '13 at 0:52
1

This decision should be a function of the platforms you envisage your users using.

In general, there are a few factors that should definitely be considered (and here I'm limiting my comments to iOS devices, but they apply even more dramatically to Android devices):

  1. Memory constraints

    iOS devices vary dramatically in available memory. In terms of just devices that support iOS 6, the iPhone 3GS and 4th generation iPod touch has 256 MB vs 1GB in the iPhone 5 and 3rd or 4th generation iPads. For that reason, you should test your application's responsiveness and other usability factors on at least three models (one 256 MB model, one 512 MB model, one 1GB model).

  2. Screens

    Non-retina iPhones (e.g. the iPhone 3GS) are very different quality screens than more modern, retina displays (e.g. the iPhone 4, 4S and 5). Likewise, the change in physical dimensions for the iPhone 5 can represent a dramatic change in some apps (especially ones intended to be used one-handed).

    The issue is even more pronounced between iPhones and iPads, where elements naturally need to be repositioned, resized and realigned to look right on the larger form factor. And again, you need to check the behaviour on a Retina and non-Retina iPad, and you should probably check an iPad mini too.

  3. Connectivity

    Depending on your app, you'll probably need to be able to test the behaviour in three states: a strong Internet connection, a dodgy (high latency, low speed) mobile connection and no Internet connection.

Of course, it's not always feasible to user test your app in every scenario (although your developers really should be testing on all target hardware since the iPhone simulator is not a reliable test). The only way to narrow down the list is to start doing some research into what devices you expect your users to use. If your app is a mobile navigation solution for drivers, you should probably prioritise phones over iPads, and it's reasonable to test on devices with functional GPS units. If you're building a bleeding edge game with incredible graphics, you probably aren't going to be targeting the iPhone 3GS anyway, so prioritise your testing on the more indicative devices.

If you have a previous version of your software, you can view your download logs and prioritise testing based on the actual devices your users use.

||improve this answer
1

We also distribute our apps to registered testers in addition to having our own devices. This means that our users bring their own devices so we don't need to make assumptions on which device they'd have.

For continuing support of devices and OS, we use analytics on existing apps and mobile websites to provide stats on OS and Devices used.

||improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.