15

I'm working on a survey platform, and I am wondering, with a design such as

different sized buttons for answers

would the differing button size for different answers introduce a bias into the responses?

Note that the survey platform has some safeguards built in to catch and eliminate responses that are likely to be fraudulent (people choosing the same option across all questions, completing much faster than most, etc.).

Any statistically-backed studies would be helpful to note.

  • 1
    You are using buttons which change their styling to represent choices? Surely radio buttons would be a better UI widget as this is perfectly suited to this selection task? – SteveD Jul 22 '16 at 11:41
  • @Splatz yes, radio buttons is another option. The hypothesis behind using buttons instead is that this would allow respondents to make a choice and advance to the next question in one click (and then navigate back if they need to change that choice). Another thing to note is that the user base of respondents traditionally is time constrained and has a low response rate, and we are trying to make responding as painless as possible. – Mike Eng Jul 22 '16 at 22:25
  • When a user navigates back to a previously answered question, the answer they gave should be indicated somehow. Radio buttons do this by default. How would you do that with buttons? – Ken Mohnkern Jul 27 '16 at 14:56
  • @KenMohnkern Good point. That previous answer could be indicated with a subtle highlight or border color that's distinct from any hover or down state on the buttons. – Mike Eng Jul 27 '16 at 15:13
  • 2
    It has been answered properly by others so I will simply leave my answer here. The answer, in image format: qph.ec.quoracdn.net/… – the other one Aug 2 '16 at 9:03
3
+50

EDIT : I found a very interesting study on the subject, it is quite long but I think it will answer most of your questions, since they have very precise number of respondent or non-respondent depending on the design choices. While it may not be directly related to button sizes, which is very specific, I strongly advise you to read to make your survey better : http://www.jwalkonline.org/docs/Grad%20Classes/Fall%2007/Cog%20Surv/class%2011/Couper%20Traugott%20Lamais%202001.pdf

And here is a solid website written by searchers that will definitely help you with some concepts about perception, cognitive aspect, etc : http://lap.umd.edu/survey_design/theory.html

I've partially readen both and I'm sure that you can come to a clear design that will suits your needs, especially with the first link. The second link is quite interesting as well but refers to more general concepts about perception.

Original answer :

I refer you to Fitts Law https://en.wikipedia.org/wiki/Fitts%27s_law for that matter. To make it short, it tells you that the difficulty to reach an element is determined by the width of the element and the horizontal distance separing it from where you are. The further you are, the harder it gets, and the wider you are, the easier it gets.

You told on your question and your comments that users want to go the painless way. So obviously some users are going to click on the easiest element, that means the wider. And anyway, making an element bigger will definitely pull the user's attention on it, and it will definitely bias the answer. Fitts Law provides you mathematical elements, so what I would do is making the three buttons the same size, and instead of aligning them horizontally I would align them vertically.

Because that way, the difficulty is the same for the three elements, since their vertical spacing is very little (so less difficulty) and with their width the same you don't over emphasize one choice over another. Like this : enter image description here

EDIT : I forgot to mention it, but you should use the same font size for the three options. It has been proven that people less read the smaller things on a web page.

  • Fitts's Law is helpful to consider. However, I don't think it answers the question without making other assumptions. Fitts's Law is about time it takes to move an input device to a target (given distance to the target and size of the target, as you stated). From time taken, you are deducing difficulty, and from difficulty, you are concluding that it will impact the rate of choices relative to each other. Those are not unreasonable to deduce in the absence of other evidence, but there are other factors like the cognitive dissonance respondents feel when accidentally making the wrong choice. – Mike Eng Aug 3 '16 at 11:00
  • I agree with you that there is other factors else than difficulty and size. But concerning only the buttons, it's important that they have the same affordance, and by putting all the characteristics to the same value (like difficulty, size, font size, color, shape, etc), you will create an uniform set. The result is that the brain won't be influenced by apparence during the "first quick scan" of the page, and user will need to dig deeper into the information shown, and have to read button labels. Concerning the reactions and the mecanisms to recover from an error, it's a whole different topic. – ArkDeus Aug 3 '16 at 12:18
  • 1
    But how does stacking the buttons vertically improve the reachability (is that even a word?) of the last element, over leaving them one next to the other? It might require extra scroll, and in certain circumstances the last button could be hidden and you could be brought to believe that the only options are YES and NO. – mjsarfatti Aug 3 '16 at 12:32
  • Of course every buttons should be visible without having to scroll, but the same problem happens if there is too much horizontal element on a wide screen. Concerning the vertical stacking, it is because usually button height is way smaller than button width (as you can see in my example). You just have to translate Fitts Law to a vertical line, and if you have three buttons of 200 width and 40 height, you'll agree that it is easier to reach the third one if it is vertically stacked than if it is horizontally stacked. – ArkDeus Aug 3 '16 at 12:39
2

I think when you are looking at a one-off selection, the size of the button would certainly have some impact on the way a user makes the selection (think about the way home pages or landing pages on websites are designed to draw attention to certain call-to-actions). They often do this by emphasizing the size, colour or styling of these buttons, to both create visual weight and highlight the various call-to-actions (usually one or two main ones). So we can assume that this is a possible way to bias or influence choice.

However, I think when you are looking at repetitive elements, such as selections in a multiple choice questionnaire (say for example in a survey platform), the repetitive nature of the styling becomes diminished to the extent that factors such as the order and label of the selections play a greater part in influencing the way users make choices.

For example, many people tend to find a way to make quick selections on a long questionnaire, and you see biases towards either selecting options at the extremes (so they don't have to move their mouse very far to click) or biased towards the middle (reference required).

2

Short answer

Yes, although the degree of bias will depend on the overall layout of the survey and the survey questions/answers.

Long answer

Do not design your survey platform so that the answers are the buttons. There are two main reasons for this:

  1. User expectation
  2. User behaviour
  3. Layout and design

1. User expectation

So, why is what people are expecting important?

Basically familiarity breeds expectation, and this usually leads to a better user experience because it matches what users are expecting. Here is some further reading to back this assertion up:

I would argue that you don't want to step away from a familiar approach unless you have something that is significantly better and therefore warrants the change. Using buttons for your answer choices will not match what users would be expecting when completing surveys or questionnaires, especially longer ones with multiple questions. As far as I can tell there is nothing to gain from using the button approach (let alone something that provides a better experience and therefore warrants the change).

2. User behaviour

Behavioural Insights research (aka behavourial economics) clearly identifies the most common behavioural traits of people. This research is basically all about helping us to understand why, when and how people make decisions.

We know from such research that one of the top 10 traits that people have is what's called a Status quo bias, also known as the 'do nothing bias'. Basically this is referring to the propensity people have for either doing nothing, maintaining the status quo, or taking the path of least resistance, rather than making a choice and taking action.

My assertion here is that using buttons will create a layout/design problem because some answers will be short (eg. yes, no, etc) and some answers may be quite long, containing an entire sentence (or even more). You can try to negate some of this (eg. ensuring all buttons are the same size), but regardless you will create an additional barrier for the user because they don't only have to read questions and try to decide on the best answer, but they also have to navigate the flow of questions and answers in a user interface that is different to the norm. This makes it harder for users to read and you run the risk of users taking the path of least resistance just to get through the questionnaire.

In other words, the risk you run is that you're creating an environment that results in the exact opposite of what you're trying to achieve. Clearly you're trying to make it easier and faster for users to navigate through and complete the questions.

3. Layout and design

The layout and design you opt for will also affect survey bias, especially since buttons are inherently an interactive object. Here is an excerpt from Fluid Surveys University:

Styling and Colouring

This section of survey bias includes any form of flare added to a survey design. It can include colour schemes, font styles, logos, videos, sounds and any other type of interactive element. Styling is important to provide stimulus to the participant and avoid respondent fatigue. Moreover, using colours and logos allows respondents to recognize a survey’s legitimacy. However, providing styling can also bias your survey. The fact is, people respond in various ways to different colours and imagery. It is important to use pretesting to ensure there are no issues with your choice of styling. Ask your pretest team whether they can clearly see and read everything in the survey and if the style used effected how they felt about the survey questions. A rule of thumb for styling is to ensure that the survey cannot be considered directed towards one demographic. Instead any added styling or colouring should make the survey look neutral while still being inviting and professional.

The reason I included the above quote is that I see some additional issues relating to the use of buttons in the manner you propose:

  • An increased likelihood of respondent fatigue
  • A design that may appear less inviting and perhaps even less professional
  • Suitability for mobile devices

This is because your proposed design is not what they will be expecting, both in terms of appearance (buttons instead of other form elements) and behaviour (auto-advancing the user through the survey). It's also likely to be harder to read for many surveys.

The excerpt above may not specifically mention smartphones or tablets, but there is no doubt that their proliferation since the onset of the iPhone in 2007 has impacted survey research. Most smartphone users now access their emails remotely which means they're much more likely to action a survey request remotely as well. This raises the following questions:

  • How will your survey come across on mobile devices?
  • How easy will it be to provide a mobile-friendly version of your survey?

All these factors will affect your completion rate (which indirectly increases bias).

Summary

If it were me, I would stick to radio buttons, check boxes, drop-down lists, etc for your answers. This is what users expect when completing a survey or questionnaire online, and using buttons has no real benefit and is more likely to corrupt your data with responses that are biased towards what was easier for users to complete.

  • Thanks. Those are useful points. One thing I would add about a potential benefit to using buttons is that it allows the user to both answer the question and advance with one click (as opposed to two using radio buttons, checkboxes, etc.). – Mike Eng Aug 3 '16 at 17:47
  • Sure, you can set it up that way, but to be honest that's also something users won't be expecting. If your survey is one that advances over multiple screens, the norm is to have a 'next' or 'continue' button at the bottom of the screen so users can progress. Having answer buttons that automatically advance to the next question will actually disorient users because it will be so unexpected. It also removes the ability for users to quickly rescan/confirm their response before proceeding and will likely result in them taking more time as they regularly go back to check their answers. Don't do it. – Monomeeth Aug 3 '16 at 19:37
  • I do applaud you for thinking outside the square and wanting to improve on the norm and trying to provide a better user experience. It's just that the 'improvement' in this case doesn't warrant the change from what users will be expecting and accustomed to. – Monomeeth Aug 3 '16 at 19:40
  • Thanks. Well, it wasn't my idea, but I'm just trying to assess it because the potential for bias was a concern that was raised in the team. Those best practices are somewhat helpful, but I'm still looking for some more concrete evidence to validate that the bias will be significant when used in a survey like this. It seems like a situation that is not very uncommon, so there must be some studies out there... – Mike Eng Aug 3 '16 at 20:15
  • You can take steps to reduce the bias by ensuring all buttons are the same width, height, colour, shape etc. The only difference is the answer text. And ensure that you take all the usual steps one would for any survey to reduce response bias (eg. wording of questions, randomisation, etc). There's a lot of research on this. Unfortunately, the button approach will add to the effort involved in reducing bias, and I agree with those in your team who raised the concerns of bias. – Monomeeth Aug 3 '16 at 22:11
0

Different sizes of buttons creates contrast. I think there's no UX principle supporting same size of buttons and it is not something really notable in the UI.

What I suggest you if you want to be sure that this aspect will not affect the choice is to find a development solutions. I understand the description for the buttons is added by the users, but I suggest to display the same value of the button for the third option, like "Both" and the description below.

The result will be three buttons, same size, with "Yes", "No", "Both".

If you choose to use checkboxes, the label size will vary anyway.

You can suggest in the admin panel short descriptions/ name, you can limit the number of characters for the labels and then make the buttons with width 33,33% of the section.

0

Don't make the input visual fluctuate depending on content.

Giving the scenario of a survey, we could assume they read before clicking. (You're not trying to get people to buy something or sign up.)

I think that on questions where the answers are similiar, the order and size of the button could influence a lazy click.

There are many articles that show the influence of button parameters: Color, text, order, size etc. Most of these are based on conversion though. An option could be a list of buttons that all have the same width & height.

But as someone else mentioned: this is one the reasons why traditional methods such as check boxes are popular.

Reading material:

0

I don't think that the size of button can change user's decision, but it depends on what the survey is about. If it's something personal, they will definitely be non-biased about it, if you just want to know about your product, that no matter what button arrangement is, user will be in a hurry to complete the survey.

I think giving radio buttons is a better option, as you mentioned that they can go back and change the choices as well.

Also, it's not a good option to take them forward on selecting their choice, user should click next to move forward, in my opinion.

0

Well,

It's pretty obvious that buttons can bias the user's actions based on their properties. Colors are the major players in C2A (Call to action) business. A lot of research has been done in order to figure out the best possible combination. On the other hand, size, shape, and position are also very important. I couldn't find the exact research paper but there are few articles explaining the various effects of these attributes. The closest I could get to was the fitts law -

https://en.wikipedia.org/wiki/Fitts%27s_law

It's mathematically proven & Tested for the position of a button. It's equally applicable to the size as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.