How do I measure a user's seriousness while filling an online form?

Question

I'm going to measure a user's seriousness while filling a form. One example of a way to do this is with a time calculation from when the survey starts until it is submitted. By considering this time, we can eliminate some users who finish the survey too quickly. I might also record the user actions (clicks and mouse movements) along with the time stamp while answering the questions.

I need to eliminate the users who will submit the form without providing serious answers. Are there any techniques to measure a user's seriousness while completing a form?

Limitations:

• Calculation can be done until the user clicks the submit button.
• We are not considering the answers.

User Seriousness

Some users are filling the form without understanding or reading the questions properly. They just fill it.

Questions can be single choice questions, multiple choice questions, open ended questions and etc.

Answer 1

I think it there's no 100% method to determine was a user serious upon filling the form. A user may fill some fields seriously, but some fields - not seriously. Moreover, some fields are more important, some - less.

For every field on your form define an importance level (1 - less important, 2 - more important, 3 - even more important, etc) for this field, and a time period to fill this field. Record time on entering and exiting for each field. If user took less time than expected to fill a field, mark this field as suspicious.

Assume we have a TextArea field (comments) in our form. And a user entered manually 150 symbols to this field just for 3 seconds. It's time to mark this field as suspicious.

And we have text passportNumber field, and user entered value manually for 1 second. Take in account copy/paste actions.

• comments has importance weight 1 and is suspicious.
• passportNumberhas importance level: 10 and is suspicious.

If you have a bunch of non-important suspicious fields, but all important fields are not suspicious, then you can consider your form as filled by a serious user. In contrary, if you have too many suspicious and important fields, then your form filled by a non-serious user.

Answer 2

One thing you have to take into consideration is if parts of the form is auto-populated, i.e. when asking for basic information such as name, email etc. All these fields might be filled out instantly and at the same time. So, in order to measure the way you want, do not ask fot that kind of "standard information".

And finally but perhaps most important of all, you have to define what seriousity is so that can be measured.

Good luck!

Answer 3

I think you should combine 2 measures: subjective and objective. You should then make analysis based on both. Here is a proposition:

Subjective questionnaire - post task

After the users fill in the forms, just ask them whether the answers they gave were correct. If the user says, "No, I just filled them to get rid of the task," ask him/her to specify which questions were not valid answers. This will provide a base for the analysis.

Objective measures - time between answers of different questions

You should not only record the time to complete the whole questionnaire, but the time between answering one question and the time to answer the next one. Here you can identify what window of time users typically need to answer an question.

Analysis

When you got the 2 types of data, you can make an analysis on how much time the users need to give a valid answer. If that time is less than a certain amount, that question is probably invalid. For example, in the end you can say if the user spends less than 3 seconds on an answer its probably an invalid or not serious answer.

Answer 4

Your definition of seriousness is actually quite hard to measure, since you are defining it as whether the user has read the question properly or understood it.

It is almost impossible to know whether someone has read the question properly unless the only way you can complete the question correctly is by following the question exactly and interpreting it correctly. Even then someone might just get lucky and happen to fill it in correctly (or auto-fill might do a very good job). Also, the only way you can test if someone has understood the question is if the question is so hard that there's no way they can easily guess it (e.g. a multiple answer selection multiple choice question). You'll see why this is the case when you try to measure comprehension (same applies to understanding of the question) using some of the more common methods, both quantitatively and qualitatively.

1. Measuring comprehension using quantitative metrics - you can perhaps argue that for a person to read the question properly it takes a certain amount of time, so you have to define what the average reading speed for a person is. But then you have to adjust this average based on variables such as age of the person, whether English is their first language, the type of content in the question, etc. But does it mean that you should exclude someone's answer if they are below or above the average? What if they are a fast reader, or if they have done lots of similar forms before, etc.? If you have a large enough sample size, you might rule out people who are two or three time above the standard deviation, but again it gets a bit trickier than you think.
2. Measuring comprehension using semi-quantitative metrics - you can perhaps model what a typical user's behaviour when filling out an online form might be like (e.g. slower at the beginning and getting quicker towards the end as they get a bit bored, say for a long form) and look to see whether the pattern holds or deviates from it to gauge whether they have spent the same amount of time reading and completing a question (for each question type) as the other users.
3. Measuring comprehension using qualitative metrics - you need to capture quantitative metrics to do this, but you can ask the users at the end if they found the questions easy complete or easy to understand, then compare this to all other users that also thought the question was easy and compare their average time to read and complete questions. Again it is not a very accurate measure, but someone who says the questions are easy but take a long time to complete it or someone who says the questions are hard but takes very little time to complete it might be possible ones to exclude.
4. Measuring comprehension by form question design - you can easily make your analysis more confusing by inserting 'control questions', that is, questions that try to assess whether the person has answered a question of the same nature differently. This would suggest that either they are not sure or they have not read the first or second question properly, thus resulting in different answers to the same question.

What I would actually suggest if you are trying to increase the quality of answers in the online form is to look at things that are within your control, such as:

• Layout of the form: make it clean and simple so that people can read it and understand the content easily. You can also arrange the form fields so that people don't scan through it too quickly (e.g. put the label above the field rather than on the same row/line)
• Language of the form: again make it easy and simple so that people can read it and understand the content easily.
• Be careful of how you use incentives: people react different ways to incentives (and the type of incentives), so use it only when they are already prepared to do the task as an extra motivation to do it well
• Calibrate your measures: so if you are going to measure user seriousness, have a standard question item that you can use as the benchmark for each type of question.

Answer 5

One very straight-forward way to do this is akin to how StackOverflow handles the review queue. Every so often, a random "experimental" question is asked - the correct answer for which is predetermined.

If they get it correct, then you know they are serious and paying attention. If not, then perhaps you give them x number of tries before handling the non-serious user.

EDIT:

Sample from SO Triage.

Now, I don't necessarily advocate showing a message like this for every application, but it was fairly simple to click a button on their test question. The test question felt very much like the rest of the questions. While captchas aren't the most fun, if the goal is to eliminate non-serious users as OP stated, then you have to take more serious measures and make it as friendly as possible.