User Experience Stack Exchange is a question and answer site for user experience researchers and experts. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We'd like to get data and feedback from real users on a new search principle on a live page (not A/B testing because too many things were changed).

Question: How should participating users be selected?

Let them choose if they want to participate, like Google Labs did it for example? Isn't there the danger that only early adaptors will participate? (would probably distort the results) If this is a good option, should we ask the users actively (i.e. a dialog when they visit the page) or passively (i.e. a link to participate on the page)?

Assign users to the experiment randomly? Or only members? Only returning users?

Any feedback and thoughts much appreciated.

share|improve this question
up vote 3 down vote accepted

Select people who are reflective of real life users.

If you can recruit enough users it makes sense to ask them if they want to participate. I have a pet hate of sites that try to force me to fill in questionnaires.

I would recommend assigning a cross section of users that represent the potential user base, new members/existing members/returning users etc.

Would this testing be web based? or face to face?

share|improve this answer
Good thoughts, thanks. The problem is that we basically don't know who the users are except if they sign in - and signed in users are returning visitors and quite a small subset of all users (no need to sign in for normal usage). It shouldn't be a problem to recruit enough users, the website in question is pretty successful. The test would be web based on a live system. – Phil Jul 28 '11 at 13:48
Didn't answer my question 100% but still helped -> Accepted. Thanks. – Phil Aug 3 '11 at 12:37

You could run a true intent study by popping up a survey link (maybe kiss insights linked to a polldaddy survey) on your search page. Use the survey link to screen people to weed out early adopters and ensure you get a cross section of your existing user base then show them your new principle search ideas using a remote research tool like (usabilia, verifyapp).

The study will still be biased towards people who have time, more likely to fill out surveys, but your screening should weed out the fan boy and girls.

You can configure a tool like to only show to logged in users, people who have viewed the page for X seconds and a heap of other stuff. Plus you can add to the javascript on your page some call which picks up the unique id of logged in users which you can gain a whole bunch of data from.

share|improve this answer
Interesting tool, thanks. – Phil Jul 28 '11 at 13:50

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.