What are the downsides of asking your current users to participate in a study (e.g. for new functionality)?

Question

What I mean is, you rarely see companies put some form of banner out there on their homepage saying:

"Hey, we're developing some new stuff and would like our customer's opinions on it. Take a quick test!"

Usually, they do those tests behind closed veils and with specifically filtered testers (e.g. in remote testing).

So my question is, what is the primary reason for that?

---

These thoughts come to mind as arguments against it:

• Only specific users would participate and skew the perspective. The average user, who is the main buyer, will ignore it; while the power or frequent users will use the chance to complain or inject their very specific wishes.
• Your users are also accustomed to the site by now and will most probably view any new design changes as "bad" and will vote for keeping what they already know.

On the other hand, these are the things that could be said in favor of it:

• You can get insight into problems that only long-term users might face, while new users do not even know they exist.
• You build more loyalty due to the trust you display for their opinion.

---

I guess I kind of answered my own question, as it usually depends on context. You use foreign, new users to test things that are supposed to attract new paying customers, while you can ask existing users when you want to improve deeper & more complex functionality.

But it seems you never see the second case, but most always the first one.

Does anyone have further points to add to the two lists or other thoughts related to that?

Answer 1

I have seen "banners" (it could be any kind of message) like you say many times. However, due to UX evolution, this is less and less common.

First of all, if you have a prototype and want to test with users, you will probably fall in one of 2 research methods: Beta Testing or Usability Testing.

Beta testing would be more as you describe, but the correct methodology is to have a limited and controlled audience, and only by invitation. Therefore, these types of banners or notices are less common every day. I remember seeing one of these ads 1 or 2 years ago (and in fact, today someone asked a question related to the OPEN beta test), but you're right, they are almost nonexistent.

Usability testing requires an increasingly controlled group, much smaller and is carried out more privately, so, again, you will not see many companies that promote these test events.

And in addition to this, you can get usability information from automated tools!

Therefore, the more advanced things are, the less you will see companies requesting testers.

As for your points "against", I definitely see them as "in favor"

Only specific users would participate and skew the perspective. The average user, who is the main buyer, will ignore it; while the power or frequent users will use the chance to complain or inject their very specific wishes.

Yes actually. And that's great! Your users will EXACTLY tell you what they want and will take the opportunity to tell you! That's what UX research is all about: to find out what the user wants EXACTLY This kind of information is very valuable and companies can pay millions for that!

Your users are also accustomed to the site by now and will most probably view any new design changes as "bad" and will vote for keeping what they already know.

Not necessarily. As a matter of fact, people like well researched and useful features. Of course, if you shock your users with a completely new way to do things, you may find problems. That's why most big companies use an Incremental UX methodology.

Bottom Line

I really can't think of any downside. Users LOVE to speak their voice and know they are being heard. I imagine there may be some scenarios in which you may find disadvantages, but honestly I can't think of any at this time. At least, in my personal experience, asking users to evaluate our UX was always positive

Additional Reading:

Radical Redesign or Incremental Change?

The 11 Secrets of Iterative and Incremental Product Development – A lightining talk

Usability vs Beta Testing (The What & When)

Answer 2

There are a few reasons for that (and there might be many more):

1. People are lazy to fill out tests. So most of them will not be interested and it takes away their focus from the main goal of the given page. (Sign up, Get our terrific newsletter etc.)
2. Prestige: you loose a lot on that. You don't seem to be professional in what you do if you ask them on your homepage what is useful for them. (They think you should know that.) Facebook groups, direct client calls, focus groups are far better solutions for these.
3. Most visitors don't know anything about your stuff. Those guys just close a page like this.

Better to ask logged in (long-time) users: "What would be more useful for you?" and give them 2-3 answers they can click on with an extra "something else". If this last one gets the most votes then it would be important to get to know them better.