It seems to be a simple problem, but i cant figure it out

Lets say, I would like to know if there is some point to implement new feature. If we have to focus on the feature or not. Lets assume there is no possible some kind of test like questioning the users or whatever else. Its function will be easy, something like - for example "webcam for ecommerce for users that are paying premium account".

To be specific, I have 1500 premium users. I can tell "Feature is used when atleast 75% of clients use it". Great! We would like to run Fake Door test, where we implement just the button for webcam and when user click on it, we show him "we are implementing this feature right now, stay with us" or whatever else (i know, fake doors isnt the best method, but it is not the point of this). I will "test" it for 14 days. In 14 days, 350 clients will come on my site and they see this feature. 265 of clients clicks on the button.

What can I say about this feature? It seems like I can say "Yes, we have to implement it, because 75% of users will use this feature" (75% of 350 is 262.5 < 265) => H0 (Atleast 75% use this feature) seems to be ok. But it is not truth at all. Because there can be HUGE error (I tested ONLY around 23% of clients).

What I am trying to achieve is:
I would like to say - "With 95% confidence, 75% of clients will use this function, so we can implement it".

I am lost of all confidence intervals, confidence levels and sample sizes, etc etc. Can someone help me how to get the confidence step-by-step and explain me, what can I count from those numbers (1500 premium users at all, 350 users saw the feature, 265 users used the feature).

  • I might be wrong and this might not be a part of your specific question, but IMHO users checking on a feature does not mean they will use the feature. Get inputs from user whether the use case you are trying to address is helpful for them to achieve their goal – Omkar Chogale Aug 21 '19 at 12:22
  • @OmkarChogale thank you, actualy i stay in front of that. “Some users” are telling me they will use this feature and it will be helpful for them. But there is no point of implementing it, unless there will be more than 75% usage. Thats why i am trying to figure the statistics out... – piggy Aug 21 '19 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.