7

I'm currently designing an Android app and would like to do a usability study to test how the app works with the target audience.

I have a question regarding recruiting participants. Since there are so many different versions of the Android OS (Jellybean, Ice Cream Sandwich etc) & different Android phones (Samsung, HTC, etc), I'm not sure how to recruit participants.

While the app seems functional on the different versions of the Android OS, I want to make sure it works well for the participants.

My concern is if I recruit people for the study and they are not familiar with the device used in the testing, it might skew the results. I would like to know from your experience how concerned I need to be about this.

Improve this question
2
  • 1
    Are your test participants going to be using their own devices or will they be handed a test device?
    – JeffH
    May 29, 2013 at 17:02
  • 1
    Plan to use a test device. Since the app is in the early stages of development, the team is not keen on allowing the participants to download the app.
    – Tara
    May 29, 2013 at 20:18

2 Answers 2

Reset to default
2

A big part of this is: what do you need to learn? If you want to record levels of performance and factors towards that, or compare this to another (or previous version of this) app, you need to make sure your test is very repeatable. If you're looking for things that you might've overlooked, things that might not be clear, and other things that could get in the way of your apps usability, you can be a bit more rough. I don't have a lot of experience with the former situation as it isn't all that informative in forming a design. However, I can provide some ideas for the latter situation.

I'd say the best way to test your app, instead of people's ability to learn to use a different device, would be to have them use the device they're already comfortable with. This will also allow them to evaluate the app against what they already know and this might give you valuable insight into their expectations. Also, having a variety of platforms might reveal problems you'd otherwise have missed.

The reality in many situations is that a lot of people with very different levels of skill and familiarity with a platform are going to use a given app. They might've just bought their first smartphone yesterday and put this app on it. Shouldn't they be just as happy with your app as people who've been using Android since day 1?

In a test though, you need to be able to relate these varying levels of skill to your results. So, record some data that may tell you something about how familiar they are with Android, and let them perform a small and common task so you can see for yourself how handy they are with your test platform.

Improve this answer
1
  • Thanks for your response. The goal is to identify the usability issues with the app. I don't have the permission to share the app for download on their device. I will need help/ideas to find a way to handle the device diversity issue. To address the issue of OS familiarity, I was thinking of conducting two studies one with experienced android users (ppl using their phone for more than a year and other group with less than 3 months).
    – Tara
    May 29, 2013 at 23:37
2

Given your constraints, I can come up with a couple of different options for you:

  1. Use a set of static images for your usability test instead of a test device with running code. You can put them on a website and point your user to it (probably with password protection), or you could load them into the user's photo library at the beginning of the study and delete them afterwards. This isn't a perfect solution, since it loses fidelity and the photos might not scale exactly as the application would on the screen, but it allows you to let your participant use their own device.

  2. Only recruit for users who use your test device, or perhaps other devices which are very similar to it. Depending on what your test device is, this could be difficult to recruit for.

  3. Same as (2), but have a larger number of test devices than just one. Are there additional devices around that are used, say by your development or quality assurance team, that you could borrow for the purposes of the study? This will give you more options.

When I recruit for mobile device studies, I recruit out anyone who has used a smartphone or tablet for less than 6 months. With users who are learning a whole new paradigm, it's difficult to discern whether their issues are with your application or simply part of the learning curve of learning how to use a brand-new type of device. It's probably not your goal to help them through their device learning curve, and I'm not sure what you would gain by conducting two studies (as you say in your response to Koen) for novice device users and more experienced device users. If you've got time/resources to conduct two studies, I think that you're more likely to gain actionable results by focusing your efforts on something else.

Improve this answer
2
  • Thanks for your response. The initial idea we had was to call people who use the phone we use internally but I wasn't sure how others in the industry do their tests. May be we can get more devices that we can use for QA and usability tests as well. The reason for two studies (novice & experienced) is that the team wants to make sure the app is intuitive to people a wide range of audience irrespective of their experience with the device.
    – Tara
    May 30, 2013 at 16:45
  • In my opinion, the usability of the OS and the usability of the application are separate questions. The state of being a new user of the OS is one that is, by its very definition, one that is fleeting. If there is something about your specific application that means that it is extremely likely to be used by novice users (for example, a healthcare application on a mobile device that is given to healthcare workers), then there is likely some benefit to doing both studies. If your application doesn't have such an inherent usage by new users, I think your research time is better spent elsewhere.
    – nadyne
    May 30, 2013 at 20:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.