Like many people who are fond of texting, I spend a lot of time hunting for the perfect emoji. Although I’m particularly fond of the “loudly crying face” ÐÐ¯Ð¨Ð½ and fire ÐÐ¯Ð¤Ðµ emojis, it’s often hard to find an image that precisely captures how I feel, especially in the moments I need one the most; when I’m feeling exhausted or angry or insecure, I usually don’t want to spend 10 minutes scrolling neurotically through over a thousand images to determine whether an upside-down smiley face or a taco best represents my emotional state.
Thanks in part to the massive popularity of emojis, several tech companies are exploring ways not only to make finding emojis easier, but to predict which ones you may want to use. Apple recently announced that the upcoming update of its Messages app will allow you to “emojify” your texts, by suggesting that you transform individual words like “beer” or “basketball” into their emoji equivalent. Another app, Dango, is going further, trying to use deep learning to predict which emojis you want to use.
Like emoticons before them, emojis serve as valuable signifiers of tone and feeling in digital spaces — environments that obscure emotional nuance more than they encourage it. Now a tool such as Dango wants to help us use these idiosyncratic and sometimes sentimental little icons to communicate better and more expressively. By asking us to consult an algorithm to figure out how we’re feeling and what we most want to express, Dango aims to be something of an emotional oracle, a body that can intuit and translate how we want to communicate in a digital language. It’s an experience both delightful and unnerving, particularly when applied to our most intimate interactions.
Initially, Dango provided only very simple emoji predictions, such as the ones Apple plans to offer: Type “happy” and it predicted a smiling face ÐÐ¯Ð¨Ð; type “pizza” and it predicted a delicious digital slice ÐÐ¯ÐÐ¥. Although this worked well for one-word substitution, it couldn’t predict relevant emojis for phrases or sentences that were more than the sum of their parts.
“It would fail in all kinds of cases where the individual words in isolation don’t convey the full essence of the sentence,” said Xavier Snelgrove, the co-founder and chief technology officer. In order to predict emoji for a phrase like “you got it” or “see you later,” the app had to be able to understand both the combined meaning of those words and how people use the visual palette of emoji to express it and respond to it.
In order to teach Dango how people actually use emojis, its developers turned to deep learning, a type of machine learning that uses algorithms to recognize and learn from patterns in data. Dango uses a recurrent neural network — a computing system inspired by the structure of biological networks like the brain — to examine over 180 million messages containing 300 million emojis on platforms such as Instagram and Twitter. Neural networks have produced major breakthroughs in helping computers understand and translate language, and they enabled Dango to transform both words and emojis into what computer scientist Geoffrey Hinton calls “thought vectors,” numerically defined points that capture both their meanings and relationships to each other.
To predict which emojis people might want to use, the neural network first learns how to distill the English word, phrase or sentence into a representation, and then tries to determine which emojis also map nearest to that semantic space. “We’re taking techniques from machine translation, but we’re using them to translate [English] into emoji,” Snelgrove said. “If you think of emoji as another language, it’s almost the same idea.”
The Dango team created a graphic that tries to visualize this semantic mapping:
Despite their widespread usage, emojis are still sometimes dismissed by traditionalists as infantile or feminine because of their simple imagery and focus on emotion. A 2015 column in the men’s style section of The New York Times fretted about whether “grown men” should really use emoji, while other hot takes have claimed that emojis are tiny clip-art word murderers intent on killing the English language and “dragging us back to the dark ages.”
But Dango helped me understand just how little those critiques fit with my own thinking. Emojis’ popularity stems in part from their ability to mirror text with cutesy pictures but also to enhance digital interactions in ways words alone cannot. A winky face can signal sarcasm, for example, or a smiley face can blunt a potentially harsh series of words. Emojis can also serve as emotional placeholders that convey a sentiment when you have no words to say — such as a heart sent without text that nonetheless signals presence and support.
While using Dango, I was delighted to find many of these usages floating to the surface: Potentially sarcastic phrases like “yeah, sure” produced both sincere thumbs up symbols ÐÐ¯Ð¡Ð and dubious faces looking askance ÐÐ¯Ð¨Ð¢. Entering “how dare you” and “I hate you” produced not just angry faces ÐÐ¯Ð¨Ð° and middle fingers ÐÐ¯Ð¦Ð¥ but also a cactus ÐÐ¯Ðâ¡, another less-than-literal interpretation that conveys a distinct emotional prickliness. It was a usage I’d never considered before, but one that felt intuitive. Talking about “haters,” meanwhile, can evoke the nail polish emoji ÐÐ¯Ð¢Ð, which can be used to convey a sassy indifference or nonchalance, as though one were admiring her own manicure rather than listening to a naysayer.
Sometimes these intuitions feel almost magical when they touch on the perfect image, though results vary, and at other times they can seem bizarre or abstruse. Slang and subcultural references can also make their way into predictions; for example, typing “that’s none of my business,” a phrase associated with a meme in which Kermit the Frog drinks iced tea, inspires frog ÐÐ¯Ð â and teacup ÐÐ¯Ðâ¡ emojis — predictions that would seem inexplicable without cultural context.
“I think Dango can have moments of delight, though it’s also going to have a lot of head scratching. How much head scratching is probably going to determine whether people use it,” said Tyler Schnoebelen, a linguist and data scientist who has worked extensively with natural language processing and wrote part of his Stanford dissertation on emoticons. “ ‘He kicked the bucket’ gets a skull [ÐÐ¯Ð¢Ð], but what are tears-of-joy [ÐÐ¯Ð¨Ð] and fists [ÐÐ¯Ð¡Ð] doing in there?”
Interpretations and associations for different emojis can also vary widely, with different usages (or perhaps even “dialects”) emerging in different countries, subcultures or platforms. The “prayer hands” emoji ÐÐ¯Ð©Ð, for example, usually expresses gratitude for Japanese users, but is sometimes interpreted as a high five by Americans. And there’s also a technological issue: Emojis can look very different depending on where you see them. Although the Unicode Consortium decides which new emojis get added to the list, each platform is allowed to create its own renditions of each concept. The “grinning face with smiling eyes” emoji ÐÐ¯Ð¨Ð, for example, displays as a smiling face on Android but a pained, grimacing face on iOS; users sending messages between different platforms are essentially using the same word to communicate contradictory meanings. Other inconsistencies include the “cookie” emoji ÐÐ¯ÐÐº, which tragically looks like a couple of dry Saltine crackers on Samsung devices. “These sorts of usages start to break down because the synonym use of the emoji only works in one font,” says Snelgrove.
Despite their imperfections and limitations, it’s still early days for emoji prediction and emojis themselves. By offering us shortcuts through this strange, charming system of emotive ideograms, apps such as Dango strive to facilitate our emotional intelligence in online interactions and develop an artificial emotional intelligence all their own. It’s a technology that promises — however imperfectly — to give us the visual vocabulary to express what we feel in the moments when words alone fail us. Or when we want to say something without having to say anything at all.
This was an edition of If Then Next, a new column that explores how algorithms intersect with culture and our everyday lives. Got feedback, suggestions or a news tip? Leave suggestions in the comments section or tweet to me @laura_hudson.