A/B testing usability with static user count

Question

I'm new to A/B testing and I have few questions.

The situation
I would be testing an information system with no new users, so user count is more or less constant. In the system there is a big form users are filling. I won't be measuring conversion rates or something like that. The aim is to measure completion times for this form and the goal is to improve the form, so it takes less time for users to fill it.
Some users might fill this form once a moth, while some might fill it multiple times a day.

The questions

1. Do I divide users in half based on form count (so there are approximately equal count of filled forms) or based on user count (so there are approx. equal count of users in each group)?
2. Can I look at each form completion as one "instance" (instead of users) despite the fact that one user can fill multiple forms?
3. How do I calculate how long should I run the test to get statistically significant results?
   For example, I have found sample size calculator (https://www.surveysystem.com/sscalc.htm), and I enter such data:
   -Confidence Level: 95%
   -Confidence Interval: 5
   and as an output I get 384. Is 384 the count of form completions for each variant?
   Let's say, there are 70 form completions a day on average. Does that mean I have to run the test for 11 days? (The calculation is: 384/70 * 2 (multiplied by 2 as there is A and B variant)) Or should I round it up to full weeks (so 14 days in this case)?

I aplogize if my questions are very simple. I have been reading quite a lot about A/B testing, but there is usually conversion rates and I cannot seem to apply it to my situation.

Answer 1

You are on the right track, but there are some things to plan for.

Try to do control measures before you get started. These will be invaluable for segmenting your users, classifying your completion times, and are a good backup if A/B testing isn't possible or has a negative impact. This will let you know how much variance in completion time you already have and may indicate trends or correlations you need to know. (The 80/20 rule says that 80% of your completions probably come from 20% of your users. Are they the fastest times or the slowest? Do they all happen on a single day of the week? Are Monday's times different from Friday's? Etc.)

Thinking about form completion as the units you are measuring, rather than users is a good idea, but you will want to make sure that each user only gets one version of the form since switching multiple times will introduce more bias. If you are worried about impacting too many users, the two audiences don't have to be equal. A sample of 10% of your users (hopefully doing 10% of your completions) can give you results. It will take longer, but impact fewer users.

Sample size calculations are for judging how closely a random sample will match the full population. If you want to select a random sample of your users to be group B, a sample size calculator will tell you how many you need to be confident that they represent the whole. (If you have 1000 users, you only need 278 to be in group B to be 95% sure their data will be within 5% of the whole group. You only need 88 if they can be within 10%. That might be OK for completion times.)

For measuring success of the test itself you need a statistical significance calculator like this one: https://www.surveymonkey.com/mp/ab-testing-significance-calculator/

However, statistical significance only measures discrete events (i.e. conversions) not timings. That's where the control data comes in. If the previous median (or average if the data is skewed) completion time was 60 seconds, you can define a successful conversion as 59 seconds or less. Then you can put those numbers into the calc and see if you need more tests. Conversion rates very different from each other will be able to reach significance quickly, but the closer they are the longer you'll have to let them run before you declare a winner. If your change makes a form twice as fast, you'll see that quickly, but you'll have to measure for a long time to detect a 5% decrease.

Note that A/B testing will only tell you which version is faster, not which the users like better or their error rate or other things. You could optimize yourself into a form that is much faster but results in a lot more bad data being collected due to typos or other mistakes.

Answer 2

I'll be upfront: I've never conducted an A/B test myself so I'll add my suggestions here to be upvoted or downvoted as the community feels appropriate, however I feel that I understand conceptually how it is used.

I think you're on the right track with how to apply this. You'd ideally divide up your user base so that some users consistently see version A, and some consistently see version B. You wouldn't want a given user to sometimes see one version during one instance, then a different version for the next instance.

As you've mentioned, your goal, in this case, is not measuring conversion rates (i.e. how many users choose to take a certain action), but rather the efficiency with which they perform the action. So, for your case, you're correct in that measuring form completion time is probably one of the better indicators of this. If you're somehow able to verify that the forms are being completed correctly (e.g. users are not returning to correct or amend their submissions, or following up with support requests), then that could be another meaningful data point to try and collect.

You've identified meaningful differences in how your users interact with the form—some use it multiple times a day (call these "frequent users"), while others use it much less ("occasional users").

As you have already hinted, I think it's wise that you divide your users such that you have a mix of frequent users and occasional users seeing each version of the form, so you may be able to notice differences in how a version affects each type of user.

Your statistical calculations sound reasonable as well: two weeks seems like a sufficient amount of time to begin drawing upon your findings. This also gives your users who see each version to become familiar with their versions, and "settle in" to how long it now takes them to complete their version of the form.

At the end of the two weeks, you can run your analyses to try and find if one had a lower average completion time than the other, and breaking down those results by different dimensions: type of user (to see if the form works better for users that are much more proficient, or perhaps simpler for users who only use it occasionally), time since being introduced to the form (to see if people improved after getting used to the new versions), or completion error rate (if applicable, to see if one version prevented errors better than the other).