Beta testing: How should users be selected?

Question

We'd like to get data and feedback from real users on a new search principle on a live page (not A/B testing because too many things were changed).

Question: How should participating users be selected?

Let them choose if they want to participate, like Google Labs did it for example? Isn't there the danger that only early adaptors will participate? (would probably distort the results) If this is a good option, should we ask the users actively (i.e. a dialog when they visit the page) or passively (i.e. a link to participate on the page)?

Assign users to the experiment randomly? Or only members? Only returning users?

Any feedback and thoughts much appreciated.

Answer 1

Select people who are reflective of real life users.

If you can recruit enough users it makes sense to ask them if they want to participate. I have a pet hate of sites that try to force me to fill in questionnaires.

I would recommend assigning a cross section of users that represent the potential user base, new members/existing members/returning users etc.

Would this testing be web based? or face to face?

Answer 2

You could run a true intent study by popping up a survey link (maybe kiss insights linked to a polldaddy survey) on your search page. Use the survey link to screen people to weed out early adopters and ensure you get a cross section of your existing user base then show them your new principle search ideas using a remote research tool like (usabilia, verifyapp).

The study will still be biased towards people who have time, more likely to fill out surveys, but your screening should weed out the fan boy and girls.

You can configure a tool like http://www.kissinsights.com/ to only show to logged in users, people who have viewed the page for X seconds and a heap of other stuff. Plus you can add to the javascript on your page some call which picks up the unique id of logged in users which you can gain a whole bunch of data from.