Using 5 Standard Emojis for Satisfaction Survey or any Likert Scale

Question

Satisfaction surveys increasingly use emoticons, i.e. a sad frownie, neutral face and happy smiley. Some additionally use traffic-light color coding (red to green). Let’s asume there are good reasons in a scenario to use 5 instead of the usual 3 (or fewer) levels, much like in a classic Likert scale (which surveys by approval) or a semantic differential rating scale.

Is there a well tested (maybe even conventional or standardized) set of five emojis found in Unicode that

1. aligns naturally (across languages and cultures) on a qualitative scale from sad to happy when used together (relative context) and
2. has every symbol (across popular / OS-provided fonts and image sets) distinctive enough to be not misidentified when used alone (absolute context)?

I see that Emojiscore, for instance, uses 😄, 😊, 😐, 😟, 😩 (top to bottom). I would have chosen a slightly different set intuitively, e.g. 😡, ☹️, 😐, ☺️, 😍 (left to right). The linked Emojipedia articles show alternative renditions and a recent study examines how some of them are interpreted very differently.

Please note that mood surveys, like Facebook’s response additions to the Like button, are a slightly different topic: Readers select one out of a predefined set of categorical icons to represent their reaction, which usually cannot be put together on a linear scale.

Related questions

• Do emojis provide any value over emoticons? – asked in 2013 prior to Unicode standardization of Japanese telco emojis, about predefined (English) codes vs. arbitrary punctuation sequences
• How should a survey (Likert Scale) be presented in a mobile application?
• Is it better to use a Likert scale or Semantic Differential for gathering attitudes towards pages?

Answer 1

Proper Likert scales need 5 blocks and often more depending on how much nuance you'd like to capture, whether you allow a neutral point, etc. If you've only 3 blocks, you might as well label them "yes" "maybe" and "no". Likert scales work precisely because they don't have defined intermediate values, they're just a continuum between 2 definite values.

So when you use smilies/emojis or any other kind of intermediate labeling, you've pretty much defeated the purpose of using a Likert scale unless you're working with a population that has trouble with abstractions.

EDIT:

I let my personal terminology get into my response, so I have to explain a little.

Strictly speaking, a "Likert scale" is a continuum with 5 points on it, ranging from "completely agree" through "somewhat agree", "neither agree nor disagree", and "somewhat disagree" to "completely disagree", or other words to those effects. Each box is labeled.

Osgood's Semantic Differential scale, on the other hand, and again speaking strictly, is a continuum with some number of boxes on it, ranging from "Industrious" to "Lazy" (or some other similarly opposing pair). None of the intermediate boxes are labeled, only the end points.

Nominally, an Osgood scale is used to capture connotation, just as a Likert scale is nominally only for agreement/disagreement.

But as should be evident, both the Likert and Osgood scales are really hooking the same mammalian ability to perceive intermediate distinctions. Conventionally their domains of use are different, but that's only convention--they're pretty much interchangeable with the Osgood scale being less restrictive.

I call them both "Likert scales" because that's easier to say than "semantic differential scale" and I can never remember Osgood's name. I should probably call them "fuzzy" scales instead, since Lotfi Zadeh's work has popularised (fsvo "popular") that more generic term.

For any case more complex than the original Likert "agree"..."disagree" continuum, it's quite easy to produce a labeled scale that appears to, but doesn't, capture the information you want.

An annoying example of that is the common one used for brick-and-mortar stores such as Lowes or Home Depot. They usually have a question "How often do you visit your local Fubar Inc?" with the choices being something like "Every day", "Twice a week", "Once a month".... Such a scale can't capture "I practically live there if I'm doing a project but otherwise never". But an unlabeled scale (Osgood scale) at least has a fighting chance because the responder can pick a point between "every day" and "never" without worrying about mapping between perception and label.

For properly-chosen continua it doesn't matter much whether the intermediate steps are labeled as long as the labels themselves make sense in context. But if they don't (the Fubar Inc example), then unlabeled is more likely to produce a good result because respondents aren't forced to map their non-verbal perceptions onto fixed labels.

(Two years ago I did a bilingual survey in aid of keeping our local post office from being closed. I used a 11-grain Osgood scale and nobody, including Portuguese immigrants with little English who got a chuckle out of my broken Portuguese, had the slightest trouble deciding which box to X to indicate how important the post office is to them, how often they visit, proportion of business vs personal use, etc.)

Answer 2

I searched high and low for equal visual weight emoji that I could use, add mouseover effects, scale, change colour etc. but I couldn't find any, In the end I made my own likert scale font, and it works quite well, if you're still looking you can get it at https://www.cormacmaher.com/Likert-Scale-icon-font-Hospitality-sector

Answer 3

Since you are dealing with Emoji you will need to consider the technical requirements of render, the liscencing, and aesthetics of the icon set you choose. Definately don't rely on the default OS rendered characters, use your own images!

For instance, EmojiOne is free and creative commons liscenced, which is great, but you need to decide if the look is appropriate for the subjects you are surveying about. I would be concerned if these emoji were used across a wide range of dynamic questions.

The emotional weight of these pictographs go beyond a clear symbolic meaning like letters or numerals. Meaning the options are not equidistant from each other, and have no clear quantifiable value in the respondents mind. Also, the visual characteristics of (for instance) 😍 may unexpectedly skew results away from that option, when they would have otherwise clicked a "5".

Also, you have to consider the layers of meaning implied by presence or absence skin color of the emoji.

In addition, I would only consider this a "gimmick" for certain apps and use cases that are exceptionally focused on technologically progressive, mobile-first, users.

A useful experiment would be to run the same survey questions across a broad audience and compare a 1-5 and 5 emoji scale and look for statistical anomalies in the results compared.