3

It seems to be a simple problem, but i cant figure it out

Lets say, I would like to know if there is some point to implement new feature. If we have to focus on the feature or not. Lets assume there is no possible some kind of test like questioning the users or whatever else. Its function will be easy, something like - for example "webcam for ecommerce for users that are paying premium account".

To be specific, I have 1500 premium users. I can tell "Feature is used when atleast 75% of clients use it". Great! We would like to run Fake Door test, where we implement just the button for webcam and when user click on it, we show him "we are implementing this feature right now, stay with us" or whatever else (i know, fake doors isnt the best method, but it is not the point of this). I will "test" it for 14 days. In 14 days, 350 clients will come on my site and they see this feature. 265 of clients clicks on the button.

What can I say about this feature? It seems like I can say "Yes, we have to implement it, because 75% of users will use this feature" (75% of 350 is 262.5 < 265) => H0 (Atleast 75% use this feature) seems to be ok. But it is not truth at all. Because there can be HUGE error (I tested ONLY around 23% of clients).

What I am trying to achieve is:
I would like to say - "With 95% confidence, 75% of clients will use this function, so we can implement it".

I am lost of all confidence intervals, confidence levels and sample sizes, etc etc. Can someone help me how to get the confidence step-by-step and explain me, what can I count from those numbers (1500 premium users at all, 350 users saw the feature, 265 users used the feature).

  • I might be wrong and this might not be a part of your specific question, but IMHO users checking on a feature does not mean they will use the feature. Get inputs from user whether the use case you are trying to address is helpful for them to achieve their goal – Omkar Chogale Aug 21 '19 at 12:22
  • @OmkarChogale thank you, actualy i stay in front of that. “Some users” are telling me they will use this feature and it will be helpful for them. But there is no point of implementing it, unless there will be more than 75% usage. Thats why i am trying to figure the statistics out... – piggy Aug 21 '19 at 12:28
3

From your example, let's imagine you have run the fake door test, which resulted in 265 visitors clicking on it, where total visitors that day were 350.



The real question (as you rightfully describe) is; between which bounds this proportion (76% (265/350)) would be for your total population (i.e. your 1500 premium subscribers). And which bounds are the closest together to still be able to say it is e.g. 95% certain.

You can calculate this.

Step 1:

Calculate the margin of error (MoE)



These are the values we need:

  • p_hat (which is just the 76% proportion found in your sample) = .76

  • alpha (this is just 1-the confidence level, the conf. level is in your description chosen to be .95 (this is very common btw) = 1 -.95 = .05

  • Critical Z value for alpha/2 (you can just google the value for this, use a look-up table, or use excel NORM.S.INV function, this does not depend on your data) = 1.96
  • n (number of participant in your sample) = 350

Now use the values from above (.76 and 0.05 and 1.96 and 350) in the following calculation;

MoE = Z(alpha/2) * square root of (p_hat * ((1 - p_hat)/n))

Keep in mind the brackets here. Filled in it looks likes this for your example:
= 1.96 * square root of (.76 * (1-.76))/350))
= 1.96 * square root of (.76 * .24)/350)
= 1.96 * square root of (.1824 /350)
= 1.96 * square root of (.00052114)
= 1.96 * 0.022828
= ~0.045

Step 2

Calculate the upper and lower limit using the MoE.
Lower limit of your 95% confidence interval = p+hat - MoE = .76 - .045 = .72 (= '72%' )
Upper limit of your 95% confidence interval = p+hat + MoE = .76 + .045 = .74 (= '81%' )

-> With 95% confidence, between 72% and 81% of visitors will click on this function.

(Of course there are some caveats, e.g. are the 350 visitors you mention from the test ('sample') all premium users? Otherwise you can't extrapolate to the confidence interval towards the 1500 number. The percentage of random visitors would be equally distributed between the 350 and the 1500.)


I hope this helps!

| improve this answer | |
1

The test that you described is not a true experiment or test. The hypothesis is unclear, the results cannot be reproduced or generalized. The method is more of a user survey, so there's no point in trying to calculate probability and CI. However, you CAN get quantitative results if you correlated the number of users who clicked and the number of them who actually signed up for the feature.

But in order to do that, you'd actually have to enable the feature. Perhaps you could turn this into a survey by enabling users to sign up for notifications about this new feature, on the fake door page e.g. "Notify me when this feature is released."

You should be able to capture some useful information there.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.