User Experience Stack Exchange is a question and answer site for user experience researchers and experts. It only takes a minute to sign up.
Sign up to join this community
I'm working on a survey platform, and I am wondering, with a design such as
would the differing button size for different answers introduce a bias into the responses?
Note that the survey platform has some safeguards built in to catch and eliminate responses that are likely to be fraudulent (people choosing the same option across all questions, completing much faster than most, etc.).
Any statistically-backed studies would be helpful to note.
EDIT : I found a very interesting study on the subject, it is quite long but I think it will answer most of your questions, since they have very precise number of respondent or non-respondent depending on the design choices. While it may not be directly related to button sizes, which is very specific, I strongly advise you to read to make your survey better : http://www.jwalkonline.org/docs/Grad%20Classes/Fall%2007/Cog%20Surv/class%2011/Couper%20Traugott%20Lamais%202001.pdf
And here is a solid website written by searchers that will definitely help you with some concepts about perception, cognitive aspect, etc : http://lap.umd.edu/survey_design/theory.html
I've partially readen both and I'm sure that you can come to a clear design that will suits your needs, especially with the first link. The second link is quite interesting as well but refers to more general concepts about perception.
Original answer :
I refer you to Fitts Law https://en.wikipedia.org/wiki/Fitts%27s_law for that matter. To make it short, it tells you that the difficulty to reach an element is determined by the width of the element and the horizontal distance separing it from where you are. The further you are, the harder it gets, and the wider you are, the easier it gets.
You told on your question and your comments that users want to go the painless way. So obviously some users are going to click on the easiest element, that means the wider. And anyway, making an element bigger will definitely pull the user's attention on it, and it will definitely bias the answer. Fitts Law provides you mathematical elements, so what I would do is making the three buttons the same size, and instead of aligning them horizontally I would align them vertically.
Because that way, the difficulty is the same for the three elements, since their vertical spacing is very little (so less difficulty) and with their width the same you don't over emphasize one choice over another. Like this :
EDIT : I forgot to mention it, but you should use the same font size for the three options. It has been proven that people less read the smaller things on a web page.
I think when you are looking at a one-off selection, the size of the button would certainly have some impact on the way a user makes the selection (think about the way home pages or landing pages on websites are designed to draw attention to certain call-to-actions). They often do this by emphasizing the size, colour or styling of these buttons, to both create visual weight and highlight the various call-to-actions (usually one or two main ones). So we can assume that this is a possible way to bias or influence choice.
However, I think when you are looking at repetitive elements, such as selections in a multiple choice questionnaire (say for example in a survey platform), the repetitive nature of the styling becomes diminished to the extent that factors such as the order and label of the selections play a greater part in influencing the way users make choices.
For example, many people tend to find a way to make quick selections on a long questionnaire, and you see biases towards either selecting options at the extremes (so they don't have to move their mouse very far to click) or biased towards the middle (reference required).
Short answer
Yes, although the degree of bias will depend on the overall layout of the survey and the survey questions/answers.
Long answer
Do not design your survey platform so that the answers are the buttons. There are two main reasons for this:
1. User expectation
So, why is what people are expecting important?
Basically familiarity breeds expectation, and this usually leads to a better user experience because it matches what users are expecting. Here is some further reading to back this assertion up:
I would argue that you don't want to step away from a familiar approach unless you have something that is significantly better and therefore warrants the change. Using buttons for your answer choices will not match what users would be expecting when completing surveys or questionnaires, especially longer ones with multiple questions. As far as I can tell there is nothing to gain from using the button approach (let alone something that provides a better experience and therefore warrants the change).
2. User behaviour
Behavioural Insights research (aka behavourial economics) clearly identifies the most common behavioural traits of people. This research is basically all about helping us to understand why, when and how people make decisions.
We know from such research that one of the top 10 traits that people have is what's called a Status quo bias, also known as the 'do nothing bias'. Basically this is referring to the propensity people have for either doing nothing, maintaining the status quo, or taking the path of least resistance, rather than making a choice and taking action.
My assertion here is that using buttons will create a layout/design problem because some answers will be short (eg. yes, no, etc) and some answers may be quite long, containing an entire sentence (or even more). You can try to negate some of this (eg. ensuring all buttons are the same size), but regardless you will create an additional barrier for the user because they don't only have to read questions and try to decide on the best answer, but they also have to navigate the flow of questions and answers in a user interface that is different to the norm. This makes it harder for users to read and you run the risk of users taking the path of least resistance just to get through the questionnaire.
In other words, the risk you run is that you're creating an environment that results in the exact opposite of what you're trying to achieve. Clearly you're trying to make it easier and faster for users to navigate through and complete the questions.
3. Layout and design
The layout and design you opt for will also affect survey bias, especially since buttons are inherently an interactive object. Here is an excerpt from Fluid Surveys University:
Styling and Colouring
This section of survey bias includes any form of flare added to a survey design. It can include colour schemes, font styles, logos, videos, sounds and any other type of interactive element. Styling is important to provide stimulus to the participant and avoid respondent fatigue. Moreover, using colours and logos allows respondents to recognize a survey’s legitimacy. However, providing styling can also bias your survey. The fact is, people respond in various ways to different colours and imagery. It is important to use pretesting to ensure there are no issues with your choice of styling. Ask your pretest team whether they can clearly see and read everything in the survey and if the style used effected how they felt about the survey questions. A rule of thumb for styling is to ensure that the survey cannot be considered directed towards one demographic. Instead any added styling or colouring should make the survey look neutral while still being inviting and professional.
The reason I included the above quote is that I see some additional issues relating to the use of buttons in the manner you propose:
This is because your proposed design is not what they will be expecting, both in terms of appearance (buttons instead of other form elements) and behaviour (auto-advancing the user through the survey). It's also likely to be harder to read for many surveys.
The excerpt above may not specifically mention smartphones or tablets, but there is no doubt that their proliferation since the onset of the iPhone in 2007 has impacted survey research. Most smartphone users now access their emails remotely which means they're much more likely to action a survey request remotely as well. This raises the following questions:
All these factors will affect your completion rate (which indirectly increases bias).
Summary
If it were me, I would stick to radio buttons, check boxes, drop-down lists, etc for your answers. This is what users expect when completing a survey or questionnaire online, and using buttons has no real benefit and is more likely to corrupt your data with responses that are biased towards what was easier for users to complete.
Different sizes of buttons creates contrast. I think there's no UX principle supporting same size of buttons and it is not something really notable in the UI.
What I suggest you if you want to be sure that this aspect will not affect the choice is to find a development solutions. I understand the description for the buttons is added by the users, but I suggest to display the same value of the button for the third option, like "Both" and the description below.
The result will be three buttons, same size, with "Yes", "No", "Both".
If you choose to use checkboxes, the label size will vary anyway.
You can suggest in the admin panel short descriptions/ name, you can limit the number of characters for the labels and then make the buttons with width 33,33% of the section.
Don't make the input visual fluctuate depending on content.
Giving the scenario of a survey, we could assume they read before clicking. (You're not trying to get people to buy something or sign up.)
I think that on questions where the answers are similiar, the order and size of the button could influence a lazy click.
There are many articles that show the influence of button parameters: Color, text, order, size etc. Most of these are based on conversion though. An option could be a list of buttons that all have the same width & height.
But as someone else mentioned: this is one the reasons why traditional methods such as check boxes are popular.
Reading material:
I don't think that the size of button can change user's decision, but it depends on what the survey is about. If it's something personal, they will definitely be non-biased about it, if you just want to know about your product, that no matter what button arrangement is, user will be in a hurry to complete the survey.
I think giving radio buttons is a better option, as you mentioned that they can go back and change the choices as well.
Also, it's not a good option to take them forward on selecting their choice, user should click next to move forward, in my opinion.
Well,
It's pretty obvious that buttons can bias the user's actions based on their properties. Colors are the major players in C2A (Call to action) business. A lot of research has been done in order to figure out the best possible combination. On the other hand, size, shape, and position are also very important. I couldn't find the exact research paper but there are few articles explaining the various effects of these attributes. The closest I could get to was the fitts law -
https://en.wikipedia.org/wiki/Fitts%27s_law
It's mathematically proven & Tested for the position of a button. It's equally applicable to the size as well.
