If the sample was just a bit too small to achieve statistical significance in a remote usability study(confidence intervals too wide due to more variability than expected)

Is it acceptable to just run more users, adding a second "leg" to the study and treating them as one data set, to try and narrow the interval? (Presuming no changes to the recruiting method or the website)

Note: data collected is both binomial (task pass/fail) and continuous (task times)

  • Out of curiosity, what kind of quantitative usability studies are you doing? I ask because I've only ever done think-aloud protocols, which are qualitative and require just 5 or 6 sessions. – Ken Mohnkern Oct 25 '17 at 14:20
  • 5-6 sessions. Depends on how confident you are in consistency. We just did a test there where all 8 users passed but that is still a P value of >0.9 which isn't really enough and I warn business about extrapolating too much from the data – colmcq Oct 25 '17 at 14:22
  • 1
    @KenMohnkern We're using automated remote testing software to benchmark a site through design changes over time. So, we're capturing task success, task time, and standardized ratings (SEQ, SUS). So it's not so much about identifying issues (we've done those studies too) but measuring performance. – Greg Hamilton Oct 26 '17 at 14:51

Calculated with the Adjusted Wald Method, as N goes up the confidence intervals shrink.

Are you evaluating specific tasks as pass/fail, or doing something more involved? Are you comparing something to something else as a control? Plenty more sample size wisdom to be found at MeasuringU.

| improve this answer | |
  • Thanks! My question is, is there anything questionable at running a few more users a few weeks later and treating it as one data set? – Greg Hamilton Oct 24 '17 at 12:50
  • Hi @GregHamilton, welcome to the site! Your clarifications should be made by editing your question, not just by commenting on answers. (This answer should probably have started by comments on your question, not by asking for clarifications inside an answer.) – Tim Grant Oct 24 '17 at 19:22
  • @GregHamilton it really depends on the experiment format but if it's simple and controlled, I don't see any reason why you can't keep adding N for a few weeks, aside from any bottleneck on stakeholder decisions. As long as the important stuff doesn't change and the distribution looks normal. – Luke Smith Oct 26 '17 at 1:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.