We'd like to get data and feedback from real users on a new search principle on a live page (not A/B testing because too many things were changed).

Question: How should participating users be selected?

Let them choose if they want to participate, like Google Labs did it for example? Isn't there the danger that only early adaptors will participate? (would probably distort the results) If this is a good option, should we ask the users actively (i.e. a dialog when they visit the page) or passively (i.e. a link to participate on the page)?

Assign users to the experiment randomly? Or only members? Only returning users?

Any feedback and thoughts much appreciated.

2 Answers 2


Select people who are reflective of real life users.

If you can recruit enough users it makes sense to ask them if they want to participate. I have a pet hate of sites that try to force me to fill in questionnaires.

I would recommend assigning a cross section of users that represent the potential user base, new members/existing members/returning users etc.

Would this testing be web based? or face to face?

  • Good thoughts, thanks. The problem is that we basically don't know who the users are except if they sign in - and signed in users are returning visitors and quite a small subset of all users (no need to sign in for normal usage). It shouldn't be a problem to recruit enough users, the website in question is pretty successful. The test would be web based on a live system.
    – Phil
    Commented Jul 28, 2011 at 13:48

You could run a true intent study by popping up a survey link (maybe kiss insights linked to a polldaddy survey) on your search page. Use the survey link to screen people to weed out early adopters and ensure you get a cross section of your existing user base then show them your new principle search ideas using a remote research tool like (usabilia, verifyapp).

The study will still be biased towards people who have time, more likely to fill out surveys, but your screening should weed out the fan boy and girls.

You can configure a tool like http://www.kissinsights.com/ to only show to logged in users, people who have viewed the page for X seconds and a heap of other stuff. Plus you can add to the javascript on your page some call which picks up the unique id of logged in users which you can gain a whole bunch of data from.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.