Paying test subjects for user testing/beta testing

Question

I've read that focus groups are a poor way of conducting user research as it attracts professional focus groupers who work the focus group "circuit" for a living. This obviously has major drawbacks as you're getting "expert" opinions instead of a random sampling of typical users. What's more, if a person participates in focus groups regularly or does it for a living, then their feedback is likely to be influenced by past sessions.

But I don't see this problem as being any different for user testing. I often see ads offering money for participation in beta testing or to be a user tester. Sites like usertesting.com offer cash rewards to users who sign up to test sites ($10 per site), so presumably their test subjects are also going to be "professional" user testers who are participating in a large number of studies to earn more money. Other companies like UserZoom source the recruitment of users to "professional panel companies", whom I assume also compensate their panel members.

So is paying user testers a good practice, or does it compromise the data that you gain from it? If it's not a good practice, then what alternatives are there to get large numbers of unbiased test subjects? Would prize raffles be more effective?

I know a lot of sites simply randomly select visitors to participate in user tests. But what if you're hired to conduct usability testing for a new service that doesn't have enough traffic to recruit enough beta testers?

Answer 1

I read about a recent study according to which 90% of UI problems can be discovered by the first 5 or so test users. (I unfortunately can't remember the link, maybe someone else can) This bears out in my experience as well, but the trick is, as stated above, to make sure that the user is unfamiliar with your UI and intentions, and to keep them uncontaminated throughout the process.

Since you don't need so many, it shouldn't be difficult to find 5-10 friends of friends or parents or whatnot that you could (as stated above) simply thank and offer a nice lunch to afterwards. Actual cash payment adds unnecessary stress and expectations to the process that is not the experience your eventual regular users will be having. Ideally the test is also happening somewhere quite casual so they don't get stage fright/think of it as some sort of intelligence test and pay more attention to, say, faqs and the like, than they normally would.

Answer 2

Selection of a representative sample is a hard problem, one you can't solve with a budget the magnitude of $10 per tester anyway. This affects all polls.

What you need is actually not a sample representative of the general population. You want a sample that represents your users. That's much easier. Your sample doesn't have to be bias free - it just doesn't have to have a bias in the aspects that relate to the system under test.

Think of it this way: For how many sites, you'd need to include a chinese rice farmer who never had access to a computer before? There must be millions and millions of them, but still, you don't need them. (Sorry, Mr. Li, for stereotyping you)

So, figure out: What are potential users? How people are using - or are supposed to use - your system? Do you have use cases with typical users? pick representive fictional users from that, e.g.:

• "gadget-junkie power user"
• "luddite CEO"
• "his boss makes him and he doesn't like it"

Try to find representatives for that. It doesn't have to be a CEO, but someone impatient, not computer savvy with lots of things on his mind. (mom? Is that you?)

---

Regarding the ultimate question:

But what if you're hired to conduct usability testing for a new service that doesn't have enough traffic to recruit enough beta testers?

Just ask friends, and invite them to dinner afterwards.

Answer 3

Regarding your last question, apart from family and friends I use free classifieds services e.g. craigslist to find random potential users.

The key is to filter the interested applicants (before you call them in) by asking them some important questions (online survey or phone) that are relevant to how you would "group" your samples. For example, a question like "Have you completed an online registration before?" can be very important when you want to measure the effectiveness of your new registration flow; depending on your test objectives or intended audience, you can have your first line of filtering based on that simple question.

I like it this way in a sense that it is more personal and it also gives me a chance to promote about the service under test. This might be useful especially when you assumed it is a new service.

I typically awarded the participants with a gift card. Yes, I told them upfront.

Answer 4

In academic research, getting a set of study participants is as much of a problem, and there is no simple answer. In essence, if you offer any form of extrinsic reward, then you will attract a certain sort of people who are attracted by the rewards. If you do not offer a reward, you will only attract those people who are prepared to help you for nothing.

The only real answer is that you need to get people in whatever way works for you, but make sure that you draw conclusions based on the sample considerations.