We are going to present the option to try the Beta version of our product to a sample population but we don't know how to calculate to how many of our users should we release it? Should I use the same sample population calculation that we use for surveys? I was thinking of using this calculator based on our total population (we have this information since this is a redesign) https://www.qualtrics.com/blog/calculating-sample-size/ Is it correct that I use this calculation to define our Beta users?

How do you calculate to how many users to release your Beta version so they can try it out?

Thank you in advance,

  • 1
    I suspect that it may help to know roughly what sort of numbers you're talking about: do your existing users number in the tens, hundreds, thousands or millions? I also suspect a lot will depend on how much resource you have to handle beta feedback (you are likely to need more resource than just processing survey results). – TripeHound Sep 12 at 8:19
  • We have thousands of users in our total population. – OrangeJuice Sep 12 at 14:01

It depends on what kind of research questions you have and what kind of data you want to collect.

The Qualtrics tool is great for any kind of quantitative analysis. Think of the margin of error as a kind of sensitivity tool: if there is an issue that affects 5% of users or less, you won't be able to detect it for sure if you have a margin of error of 5%.

If you're interested in qualitative feedback, then having a sample of this size will likely generate a large amount of text to read and analyze. If you can set up a short survey to get structured feedback, that will be helpful.

Why are you talking hard numbers ?

The whole point of the Beta program is to get users to test a new version, you want them to test the program extensively and report any issues. Ideally, you would want the techier tier of your user base, because they have more experience in finding bugs, that are also fanboys, as fanboys are eager to help squash any bugs found.

Enrolling users on a Beta program because they took the time to answer a survey is, I think, getting the target population wrong.

Do as many other software companies do, offer a Beta version alongside the usual version with a message reading that the Beta version is a development version that needs to be tested, not for the faint of heart.

  • Thanks. Yes, that is what we are doing: we are releasing a beta version alongside the usual version with a message letting them try the new beta if they want to. But we dont know to how many users we should show that opt-in message. – OrangeJuice Sep 15 at 18:43

There is not a certain number of testers. You must determine the goals of your analysis. Also you must determine the planned number of users of your product. After that you can understand what types of testers you need, how many types and how many testers you need.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.