Is there a way to measure if version A or B is better based on usability?
Yes. Select what dimension of usability you want to pursue, and measure that. Usually, A/B tests measure “success” or “conversion” rates, or the probability that a user will complete a key task. So if the task is to fill out a form (e.g., to make purchase, signup, or post), count how many do and don’t complete and submit the form with each design. However, A/B testing can be used for other dimensions of usability, such as speed of task completion (e.g., time to fill out and submit a form), or proportion of users making an error (e.g., entering invalid data into a text field).
In any case, you need a reasonable theory (the “why” part) in order to interpret the results. A/B testing shouldn’t be just “let’s change a random attribute and see what happens.” Pick A and B because there’s reason to believe that one could be better than the other on your chosen dimension of usability. It doesn’t have to be a terribly sophisticated psychological theory. Maybe you think a red button is better because it would be easier to find on your form. Maybe you think a blue button is better because your users associate red with destructive dangerous actions they don’t want to do.
Without a theory, performance is ambiguous. Did a lot of users click on blue because it looks safer? Or did they do it because they mistook if for something else? The former is a success, indicating better usability. The latter is an error, indicating worse usability.
As Andrew Martin suggests in his comment, sometimes you have to run a series of A/B tests to be sure you’re interpreting it right. If you think blue performed better than red because it looked “safer,” then can you hypothesize that green would perform better still (if your users’ culture regards it as “safest”)? So now you test green versus blue. A/B testing is the scientific method at work.
Is there a way to calculate the size of a A/B test group regarding usability (without the use of conversion rates)?
No. The statistical significance of a result tends to be driven by the size of the smallest marginal, which in A/B testing is usually the total number of conversions (number of conversions from both A and B, which tend to be much less than the number of non-conversions). However, you can probably can get a reasonable estimate of the conversion rate from the current performance of the product.
You also need to define what you regard as a “substantial” difference in performance between A and B. How much better does one design have to be than the other? In the case of rates, what percent more conversions does B need to have before you care about the difference between A or B? Ten percent more conversions? (That's like 10 out of 50 visitors convert for A, but 11 out of 50 converted for B.) Fifty percent more? (10 out of 50 versus 15 out of 50.) Twice the rate? (100% more, for 10 out of 50 versus 20 out of 50.)
Once you select a substantial difference, a conservative (i.e., possibly more than enough) estimate of the number of conversions you need is:
Needed number of conversions from A and B = 4 * ( (d+2)/d )^2
Where d is your selected substantial difference expressed as a proportional difference (not percent). For example, if you care if B does as little as 10% better than A, then it’s 4 * (2.10/0.10)^2 = 1764 conversions. If you care only if B is at least twice as good as A, then it’s 4 * (3/1)^2 = 36 conversions.
Now use your estimate of the overall conversion rate to get your total sample size. If you estimate that 1 in 10 visitors convert, then your total sample size (number of visitors) is 10 * 1764 = 17640, or 10* 36 = 360, depending on your chosen substantial difference. If 1 in 3 convert, then it’s 3 * 1764 = 5292, or 3 * 36 = 108.
It’s only an estimate. However, if it ends up being not enough (there appears to be a substantial difference but it isn’t statistically significant), then simply run the A/B test for longer to get a bunch more users (e.g., double the sample size).
Stats Geek Corner
I derived the formula above from the normal approximation of the binomial distribution. When the number of conversions is much smaller than the number of non-conversions, the statistical test can conservatively be a binomial test of the difference of the conversions with null hypothesis P = 0.5, provided an equal number of users get A and B. In the normal approximation of the binomial, the standard error, se, with P = 0.5 is:
se = (P * (1-P) * N)^0.5
se = 0.5* n^0.5
Expressed as a proportion of n, the number of conversions, that’s
se = 0.5* n^0.5 / n
se = 1/(2 * n^0.5)
Applying a two-tailed z test, at the 0.05 level, the proportion of conversions for Design B (to pick one arbitrarily) has to be at least 1.96 * se over 0.5 (the null hypothesis) to achieve significance. Let’s round 1.96 to 2 (we’re only estimating), so:
sig prop for B = 0.5 + 2 * 1/(2 * n^0.5)
sig prop for B = 0.5 + 1/n^0.5
If B is 2 se over 0.5, A must be 2 under:
sig prop A = 0.5 – 1/n^0.5
so they sum to 1.
When the number of users taking A and B is equal, the proportional difference, d, is the ratio of these two proportions minus 1. For example, if 0.4 of conversions occur with A and 0.6 of conversion occur with B, then B is 0.6/0.4 -1 = 50% better than A. So:
d = sig prop B / sig prop A – 1
d = (0.5 + 1/n^0.5) / (0.5 – 1/n^0.5) - 1
d = (n^0.5 + 2) / (n^0.5 – 2) – 1
Now we solve for n:
(d + 1)*(n^0.5 – 2) = n^0.5 + 2
d*n^0.5 + n^0.5 – 2*d – 2 = n^0.5 + 2
-2*d – 4 = -d*n^0.5
2*(d + 2)/d = n^0.5
4*( (d+2)/d )^2 = n
“Quantum electrodynamics,” as the physicists say.
I ran chi-square tests of independence for d = 0.1 to 3.2, with conversion rates from 1-in-four to 1-in-1000, and the actual p-values observed with the derived sample size and proportional difference were always between 0.02 and 0.05, so as a conservative estimate, it works okay. Get down to 1 conversion in 2 (equal number of conversions and non-conversions) it gets too conservative IMO: actual p-values are less than 0.006.
More Geekiness: The Caveats
The formula is intended to be a simple back-of-the-envelope rough and conservative estimate. As you can see, there are several simplifying shortcuts to make an easy formula that doesn’t take a whole lot of computation or statistical understanding. It’s basically a statistical party trick, and a way of showing that a “big enough” sample size can varying a lot by the situation.
Furthermore, the formula gives the sample size you need for a d observed in the sample. A more sophisticated analysis would calculate the sample size for a d of the population and a selected probability of detecting any significant difference in the sample, i.e., the statistical power. For the record, the above formula gives a sample size with the power of only 0.5 for a population effect size of d. The above formula could be extended to include a selected level of power, but that adds the burden of choosing a level of power, in addition to choosing d and estimating the conversion rate (I’d also argue that. if you go that route, you may also want to reconsider alpha = 0.05).
If there are concerns about the expense of running the A/B test, then by all means optimize the sample size by performing a power analysis, one which does not have simplifying shortcuts (e.g., using G*Power or the like). Hire a statistician to do it for you, if necessary.
