Return to Answer

How can this discrepancy be reconciled?

You have divergent results because the number of participants is small and not representative. There is no randomization or blinding to prevent bias. You're also not calculating the relevant stats. (What are the standard deviation, margin of error, confidence intervals, odds ratios, p values, etc?)

Further, you appear to be doing iterative design, not "experiments". There is nothing wrong with iterative design, but the data you collect are likely meaningless beyond the current design. They cannot be used to meaningfully compare designs against each other. Even if they could, there aren't enough participants to measure the effect of small changes. But you don't need large numbers of users for iterative design. Just enough to identify problems.

In an experiment, you'd have multiple designs A/B/C... tested in parallel. Participants would be randomized to the designs (as well as task order). Experimenters would not know which design individual participants were using. Experimenters would not observe participants directly. Experimenters would pre-decide what statistical tests are appropriate. They would not begin processing data until after it had all been collected. Etc. If you were testing drugs, your methodology (as well as insufficient participants) would likely prevent FDA approval.

How could one make sense of these results?

You did a t-test and found no significant difference. The "study" is likely underpowered with only five subjects in each group. Even if you had enough numbers to demonstrate significance, the study needs to be redesigned, and the survey has to be checked for reliability and validity.

The System Usability Scale (SUS) is described by its original developer as "quick and dirty". It appears to have been validated as a global assessment, but it's probably not appropriate for comparison. Imagine there were something known as Global Assessment of Functioning that physicians used to evaluate health. Is someone with condition A and GAF 85 "healthier" than someone with condition B and GAF of 80? Does it even make sense to compare A and B this way?

Even if these problems were all addressed, you are still doing iterative design. I would expect differences between successive iterations to be non-significant. Suppose you were testing drugs. Would you expect significantly different results between 100mg and 101mg doses? What about 101mg and 102mg? Etc.

What to do... ?

Stop collecting (or misusing) quantitative data when you know you won't have the numbers to demonstrate significance. Stop having "expectations", as it is a source of bias that can lead you astray. Redesign experiments to eliminate bias when it is impossible to eliminate expectations. Understand that iterative design is not experimentation. The value of small usability reviews is to screen for problems, not confirm success or produce stats.

... it seems the confidence intervals are so wide, that the intermediate results I got should not be a reason of concern.

That is what I "expected".

How can this discrepancy be reconciled?

You have divergent results because the number of participants is small and not representative. There is no randomization or blinding to prevent bias. You're also not calculating the relevant stats. (What are the standard deviation, margin of error, confidence intervals, odds ratios, p values, etc?)

Further, you appear to be doing iterative design, not "experiments". There is nothing wrong with iterative design, but the data you collect are likely irrelevant beyond the current design. They cannot be used to meaningfully compare designs against each other. Even if they could, there aren't enough participants to measure the effect of small changes. But you don't need large numbers of users for iterative design. Just enough to identify improvements for the next iteration.

In an experiment, you'd have multiple designs A/B/C... tested in parallel. Participants would be randomized to the designs (as well as task order). Experimenters would not know which design individual participants were using. Experimenters would not observe participants directly. Experimenters would pre-decide what statistical tests are appropriate. They would not begin processing data until after it had all been collected. Etc. If you were testing drugs, your methodology (as well as insufficient participants) would likely prevent FDA approval.

How could one make sense of these results?

You did a t-test and found no significant difference. The "study" is likely underpowered with only five subjects in each group. Even if you had enough numbers to demonstrate significance, the study needs to be redesigned, and the survey has to be checked for reliability and validity.

The System Usability Scale (SUS) is described by its original developer as "quick and dirty". It appears to have been validated as a global assessment, but it's probably not appropriate for comparison. Imagine there were something known as Global Assessment of Functioning that physicians used to evaluate health. Is someone with condition A and GAF 85 "healthier" than someone with condition B and GAF of 80? Does it even make sense to compare A and B this way?

Even if these problems were all addressed, you are still doing iterative design. I would expect differences between successive iterations to be non-significant. Suppose you were testing drugs. Would you expect significantly different results between 100mg and 101mg doses? What about 101mg and 102mg? Etc. (How massive would n need to be to detect such minute differences?)

What to do... ?

Understand that iterative design is not experimentation. The value of small usability reviews is to screen for problems, not confirm success or produce stats.

Stop collecting (or "misusing") quantitative data when you know you won't have the numbers to demonstrate significance. Stop having "expectations", as it is a source of bias that can lead you astray. Redesign experiments to reduce bias.

... it seems the confidence intervals are so wide, that the intermediate results I got should not be a reason of concern.

That is as "expected".

How can this discrepancy be reconciled?

You have divergent results because the number of participants is small and not representative. There is no randomization or blinding to prevent bias. You're also not calculating the relevant stats. (What are the standard deviation, margin of error, confidence intervals, odds ratios, p values, etc?)

Further, you appear to be doing iterative design, not "experiments". There is nothing wrong with iterative design, but the data you collect are likely meaningless beyond the current design. They cannot be used to meaningfully compare designs against each other. Even if they could, there aren't enough participants to measure the effect of small changes. But you don't need large numbers of users for iterative design. Just enough to identify problems.

In an experiment, you'd have multiple designs A/B/C... tested in parallel. Participants would be randomized to the designs (as well as task order). Experimenters would not know which design individual participants were using. Experimenters would not observe participants directly. Experimenters would pre-decide what statistical tests are appropriate. They would not begin processing data until after it had all been collected. Etc. If you were testing drugs, your methodology (as well as insufficient participants) would likely prevent FDA approval.

How could one make sense of these results?

You did a t-test and found no significant difference. The "study" is likely underpowered with only five subjects in each group. Even if you had enough numbers to demonstrate significance, the study needs to be redesigned, and the survey has to be checked for reliability and validity.

The System Usability Scale (SUS) is described by its original developer as "quick and dirty". It appears to have been validated as a global assessment, but it's probably not appropriate for comparison. Imagine there were something known as Global Assessment of Functioning that physicians used to evaluate health. Is someone with condition A and GAF 85 "healthier" than someone with condition B and GAF of 80? Does it even make sense to compare A and B this way?

Even if these problems were all addressed, you are still doing iterative design. I would expect differences between successive iterations to be non-significant. Suppose you were testing drugs. Would you expect significantly different results between 100mg and 101mg doses? What about 101mg and 102mg? Etc.

What to do... ?

Stop collecting (or misusing) quantitative data when you know you won't have the numbers to demonstrate significance. The value of small usability reviews is to screen for problems, not confirm success or produce stats.

Stop having "expectations", as it is a source of bias that can lead you astray. Redesign experiments to eliminate bias when it is impossible to eliminate expectations. Understand that iterative design is not experimentation.

... it seems the confidence intervals are so wide, that the intermediate results I got should not be a reason of concern.

That is what I "expected".

How can this discrepancy be reconciled?

You have divergent results because the number of participants is small and not representative. There is no randomization or blinding to prevent bias. You're also not calculating the relevant stats. (What are the standard deviation, margin of error, confidence intervals, odds ratios, p values, etc?)

Further, you appear to be doing iterative design, not "experiments". There is nothing wrong with iterative design, but the data you collect are likely meaningless beyond the current design. They cannot be used to meaningfully compare designs against each other. Even if they could, there aren't enough participants to measure the effect of small changes. But you don't need large numbers of users for iterative design. Just enough to identify problems.

In an experiment, you'd have multiple designs A/B/C... tested in parallel. Participants would be randomized to the designs (as well as task order). Experimenters would not know which design individual participants were using. Experimenters would not observe participants directly. Experimenters would pre-decide what statistical tests are appropriate. They would not begin processing data until after it had all been collected. Etc. If you were testing drugs, your methodology (as well as insufficient participants) would likely prevent FDA approval.

How could one make sense of these results?

You did a t-test and found no significant difference. The "study" is likely underpowered with only five subjects in each group. Even if you had enough numbers to demonstrate significance, the study needs to be redesigned, and the survey has to be checked for reliability and validity.

The System Usability Scale (SUS) is described by its original developer as "quick and dirty". It appears to have been validated as a global assessment, but it's probably not appropriate for comparison. Imagine there were something known as Global Assessment of Functioning that physicians used to evaluate health. Is someone with condition A and GAF 85 "healthier" than someone with condition B and GAF of 80? Does it even make sense to compare A and B this way?

Even if these problems were all addressed, you are still doing iterative design. I would expect differences between successive iterations to be non-significant. Suppose you were testing drugs. Would you expect significantly different results between 100mg and 101mg doses? What about 101mg and 102mg? Etc.

What to do... ?

Stop collecting (or misusing) quantitative data when you know you won't have the numbers to demonstrate significance. Stop having "expectations", as it is a source of bias that can lead you astray. Redesign experiments to eliminate bias when it is impossible to eliminate expectations. Understand that iterative design is not experimentation. The value of small usability reviews is to screen for problems, not confirm success or produce stats.

... it seems the confidence intervals are so wide, that the intermediate results I got should not be a reason of concern.

That is what I "expected".

How can this discrepancy be reconciled?

You have divergent results because the number of participants is small and not representative. There is no randomization or blinding to prevent bias. You're also not calculating the relevant stats. What are the standard deviation, margin of error, confidence intervals, odds ratios, p values, etc etc?

How could one make sense of these results?

You did a t-test and found no significant difference. The "study" is likely under powered with only five subjects in each group. Even if you had enough numbers to demonstrate significance, the study needs to be redesigned, and the survey has to be checked for reliability and validity.

What to do... ?

Stop collecting quantitative data when you know you won't have the numbers to demonatrate significance. The value of small usability reviews is to screen for problems, not confirm success or produce stats.

Stop having "expectations", as it is a source of bias that can lead you astray.

How can this discrepancy be reconciled?

You have divergent results because the number of participants is small and not representative. There is no randomization or blinding to prevent bias. You're also not calculating the relevant stats. (What are the standard deviation, margin of error, confidence intervals, odds ratios, p values, etc?)

Further, you appear to be doing iterative design, not "experiments". There is nothing wrong with iterative design, but the data you collect are likely meaningless beyond the current design. They cannot be used to meaningfully compare designs against each other. Even if they could, there aren't enough participants to measure the effect of small changes. But you don't need large numbers of users for iterative design. Just enough to identify problems.

In an experiment, you'd have multiple designs A/B/C... tested in parallel. Participants would be randomized to the designs (as well as task order). Experimenters would not know which design individual participants were using. Experimenters would not observe participants directly. Experimenters would pre-decide what statistical tests are appropriate. They would not begin processing data until after it had all been collected. Etc. If you were testing drugs, your methodology (as well as insufficient participants) would likely prevent FDA approval.

How could one make sense of these results?

You did a t-test and found no significant difference. The "study" is likely underpowered with only five subjects in each group. Even if you had enough numbers to demonstrate significance, the study needs to be redesigned, and the survey has to be checked for reliability and validity.

The System Usability Scale (SUS) is described by its original developer as "quick and dirty". It appears to have been validated as a global assessment, but it's probably not appropriate for comparison. Imagine there were something known as Global Assessment of Functioning that physicians used to evaluate health. Is someone with condition A and GAF 85 "healthier" than someone with condition B and GAF of 80? Does it even make sense to compare A and B this way?

Even if these problems were all addressed, you are still doing iterative design. I would expect differences between successive iterations to be non-significant. Suppose you were testing drugs. Would you expect significantly different results between 100mg and 101mg doses? What about 101mg and 102mg? Etc.

What to do... ?

Stop collecting (or misusing) quantitative data when you know you won't have the numbers to demonstrate significance. The value of small usability reviews is to screen for problems, not confirm success or produce stats.

Stop having "expectations", as it is a source of bias that can lead you astray. Redesign experiments to eliminate bias when it is impossible to eliminate expectations. Understand that iterative design is not experimentation.

... it seems the confidence intervals are so wide, that the intermediate results I got should not be a reason of concern.

That is what I "expected".