Is testing with only 1000 user in AB testing, can give us a real data? or it will just be a waste of time?
1
-
It is depends on the all user count and many things... Testing with just 1000 users is a joke for big site like Facebook. In some cases, even 10 users also enough.– I am the Most Stupid PersonFeb 23, 2018 at 10:50
Add a comment
|
1 Answer
Where Option A and Option B have markedly different response rates, 1000 is enough.
Popular A/B test platform Optimizely, explains it this way: https://help.optimizely.com/Analyze_Results/Statistical_significance_in_Optimizely
The big selling point of A/B test platforms is that they provide "statistically significant" results, but "statistically significant" is a relative concept. For example, Optimizely sets significance at 90%. In practice, where tested variations are minor, you may need 3000 to 5000 clicks on each variant to reach the 90% threshold and declare a winner.
BTW a 90% threshold is OK for marketing, but most scientific experiments set their mark at 95% or 98%.
Fortunately UX is not science. Most of our design debates can be settled with as few as five test subjects. See NNG's famous article on small-batch testing: https://www.nngroup.com/articles/how-many-test-users/
-
1NNG's 5 people testing (debatable) is for usability testing, not A/B. I would also argue with the set number of 1000, unless the product/service is addressed to an homogeneous target with homogeneous variables. In which case, it could be even lower than 1000. With heterogeneous variables (for example, age or geo) I'd go with a bigger number if possible– DevinFeb 23, 2018 at 19:48