A big part of this is: what do you need to learn? If you want to record levels of performance and factors towards that, or compare this to another (or previous version of this) app, you need to make sure your test is very repeatable. If you're looking for things that you might've overlooked, things that might not be clear, and other things that could get in the way of your apps usability, you can be a bit more rough. I don't have a lot of experience with the former situation as it isn't all that informative in forming a design. However, I can provide some ideas for the latter situation.
I'd say the best way to test your app, instead of people's ability to learn to use a different device, would be to have them use the device they're already comfortable with. This will also allow them to evaluate the app against what they already know and this might give you valuable insight into their expectations. Also, having a variety of platforms might reveal problems you'd otherwise have missed.
The reality in many situations is that a lot of people with very different levels of skill and familiarity with a platform are going to use a given app. They might've just bought their first smartphone yesterday and put this app on it. Shouldn't they be just as happy with your app as people who've been using Android since day 1?
In a test though, you need to be able to relate these varying levels of skill to your results. So, record some data that may tell you something about how familiar they are with Android, and let them perform a small and common task so you can see for yourself how handy they are with your test platform.