We'd like to get data and feedback from real users on a new search principle on a live page (not A/B testing because too many things were changed).
Question: How should participating users be selected?
Let them choose if they want to participate, like Google Labs did it for example? Isn't there the danger that only early adaptors will participate? (would probably distort the results) If this is a good option, should we ask the users actively (i.e. a dialog when they visit the page) or passively (i.e. a link to participate on the page)?
Assign users to the experiment randomly? Or only members? Only returning users?
Any feedback and thoughts much appreciated.