How do I phrase usability questions to get useful data

Question

Exec. Summary (or tldr): How do I phrase my questions to get the best data?

(This is a follow up to this question.)

I have a very long, paged/saveable form, used by banking agents to onboard businesses. The agents are proficient in a 90s version of this form already - we are upgrading the application to the 21st century.

The form has phone number fields as well as date fields. Currently, the fields are componentized: [MM] [DD] [YYYY] or [XXX] [XXX] [XXXX] Ext[XXXX]. But I'm not sure that's very efficient. It may reduce error but I suspect that it does so at great cost to speed. Remember, these are trained users. For example, it might be better to have the option to copy/paste whole strings of phone numbers, say, from a document they already have.

I think a full-blown usability study might be less useful than simply asking the users. So I'm writing up some questions in anticipation of asking them.

My question to y'all is this:

How do I phrase the question(s) to get the most informative responses?

Should I lead them toward opportunities they might not know are available, or should I not bias them, to get answers that aren't just re-enforcing my own expectations? Both have problems. Examples:

• "Considering both speed and accuracy, what way(s) for entering dates/phone numbers are/would be most efficient for you?" (Open-ended and unbiased but also not helping them explore possibilities)

• "Would the ability to copy/paste whole number strings at a time into a single field be useful?" (Helpful but leading them down a path)

• "Do separated fields provide the accuracy you need without sacrificing speed?" (neutral)

• "How do you enter these fields now?" or "How do you expect to be able to enter these fields?" (Not useful, since they don't have options or expectations in the current app.)

Of course I also don't want to inundate them with questions, so I have to be judicious with them.

How would you handle this?

Answer 1

The information you need to solve this problem is not if copy-pasting is faster and more accurate than typing: very likely, it is and it doesn't depend on user preferences.

The information you need is what proportion of data entry the users do based on digital documents they can copy-paste from, vs analog documents / phone calls, etc. They might have an image scan of the client ID with no OCR: they won't be able to copy a date from it.

These are questions you can ask in your survey: (to be adapted to the specificity of the user role)

• In the last 24h, how many times did you enter data using this form?
• Please select all the sources that you used in the last 24h to collect the data: [] paper ID [] oral answers to my question [] image scan [] online form [] pdf or word doc [] other

For each option selected, ask the follow-up question: - in the last 24h, how many times did you enter data based on [option]?

Then you can calculate the proportion across all users.

Why ask the number of times in the last 24h rather than directly asking for the proportion?

Because humans are notoriously bad at estimates, and even worse at making a calculation of proportion. Working memory is very short term, so longer than 24h will give you unreliable answers. The last 24h might not be representative for any particular individuals, but the differences will average out across a large number of respondents.

Answer 2

Asking your users questions about how they would hypothetically use the new form design is a bit tricky. It's a bit difficult for them to give an accurate answer, because they have to picture the interface in their heads.

You could have them to using various prototypes of the form inputs and get their feedback as they work through it, or listen in on agent trainer sessions to find out how they're training the agents to use these forms.

To find out how users are making use of the forms, you could use a screen recorder and observe them remotely or next to them if possible.

But if you absolutely had to only do a survey, I would re-phrase ""Considering both speed and accuracy, what way(s) for entering dates/phone numbers are/would be most efficient for you?" " as " 1. How do you usually enter phone numbers? [Type phone number into form] [Copy and paste from existing document] [Other - open field]

2.1 Which way of entering dates do you prefer" [show single field] [show split fields]

2.2 Why do you choose this? [This is a more accurate method] [This is a faster method] [Other - open field]

By leaving out the value judgments in the question itself (we assume users want both speed and accuracy) and splitting up the answers into results that you can easily analyze (plus easier for participants to answer)

Answer 3

The way you have the questions phrased I would not ask any of them directly to people. You have phrased them more like research questions which you could answer yourself after making some careful observations of what the people actually do in real-life or experimental scenarios.

If you ask people hypothetical questions you’ll get non-data.