1

If the sample was just a bit too small to achieve statistical significance in a remote usability study(confidence intervals too wide due to more variability than expected)

Is it acceptable to just run more users, adding a second "leg" to the study and treating them as one data set, to try and narrow the interval? (Presuming no changes to the recruiting method or the website)

Note: data collected is both binomial (task pass/fail) and continuous (task times)

Improve this question
3
  • Out of curiosity, what kind of quantitative usability studies are you doing? I ask because I've only ever done think-aloud protocols, which are qualitative and require just 5 or 6 sessions.
    – Ken Mohnkern
    Commented Oct 25, 2017 at 14:20
  • 5-6 sessions. Depends on how confident you are in consistency. We just did a test there where all 8 users passed but that is still a P value of >0.9 which isn't really enough and I warn business about extrapolating too much from the data
    – colmcq
    Commented Oct 25, 2017 at 14:22
  • 1
    @KenMohnkern We're using automated remote testing software to benchmark a site through design changes over time. So, we're capturing task success, task time, and standardized ratings (SEQ, SUS). So it's not so much about identifying issues (we've done those studies too) but measuring performance.
    – Greg Hamilton
    Commented Oct 26, 2017 at 14:51

1 Answer 1

Reset to default
1

Calculated with the Adjusted Wald Method, as N goes up the confidence intervals shrink.

Are you evaluating specific tasks as pass/fail, or doing something more involved? Are you comparing something to something else as a control? Plenty more sample size wisdom to be found at MeasuringU.

Improve this answer
3
  • Thanks! My question is, is there anything questionable at running a few more users a few weeks later and treating it as one data set?
    – Greg Hamilton
    Commented Oct 24, 2017 at 12:50
  • Hi @GregHamilton, welcome to the site! Your clarifications should be made by editing your question, not just by commenting on answers. (This answer should probably have started by comments on your question, not by asking for clarifications inside an answer.)
    – Tim Grant
    Commented Oct 24, 2017 at 19:22
  • @GregHamilton it really depends on the experiment format but if it's simple and controlled, I don't see any reason why you can't keep adding N for a few weeks, aside from any bottleneck on stakeholder decisions. As long as the important stuff doesn't change and the distribution looks normal.
    – Luke Smith
    Commented Oct 26, 2017 at 1:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.