News & Technology

13 September 2017

Facebook AI learns human reactions after watching hours of Skype

AI Skype
Deadpanning won’t cut it any more

Johannes Eisele/AFP/Getty

THERE’S often something not quite right about humanoid robots. They are cute up to a point, but once they become a bit too realistic, they start to creep us out – a foible called the uncanny valley.

Now Facebook wants robots to climb their way out of it. Researchers at the company’s AI lab have developed an expressive bot, an animation of a face controlled by an artificially intelligent algorithm.

The algorithm was trained on hundreds of videos of Skype conversations, so that it could learn and then mimic how people adjust their expressions in response to each other. In tests, it successfully passed as human-like.


Non-verbal facial cues are a key part of human conversation, says Louis-Philippe Morency at Carnegie Mellon University in Pittsburgh. We use them to signal that we are listening to someone and engaging with them.

To mimic this organic feature of human communication, the algorithm divided individual faces into 68 key points that it monitored throughout each Skype conversation. Eventually, it learned to produce the same nods, blinks and small mouth movements that we involuntarily make during conversation.

The bot was then able to look at a video of a human speaking, and choose in real time what the most appropriate facial response would be. If the person was laughing, for example, the bot might choose to open its mouth too, or tilt its head subtly.

The Facebook team then tested the system with panels of people who watched animations that included both the bot reacting to a human, and a human reacting to a human. The volunteers judged the bot and the human to be equally natural and realistic.

Facebook will present the work at the International Conference on Intelligent Robots and Systems in Vancouver, Canada, later this month.

However, the animations the team used were quite basic, so it’s not clear whether a humanoid robot powered by this algorithm would have natural-seeming reactions.

Additionally, learning the basic rules of facial communication might not be enough to create truly realistic conversation partners, says Goren Gordon at Tel Aviv University in Israel. “Actual facial expressions are based on what you are thinking and feeling.”

In this case, the Facebook system ends up creating a kind of “average personality”, say Morency. In future, more sophisticated bots might be able to pick from a range of personalities or adapt their own to match that of the person they are talking to.

Robots aren’t so good at mastering these subtle elements of human interaction, says Gordon. We already know that humans prefer speaking with robots that mimic their own facial expression, he says, but now Facebook is trying to take robot conversations to the next level. “At some point we’ll get out of the uncanny valley and come out at the other side,” he says.

This article appeared in print under the headline “Chatbot learns apt facial expressions”