How Facebook’s Machines Got So Good At Recognizing Your Face

In DeepFace, Facebook has some of the best facial recognition technology in the world. Here’s why it’s a big deal.

How Facebook’s Machines Got So Good At Recognizing Your Face
[Image: Flickr user Justin Pickard]

The ability to recognize human faces has always been a benchmark for artificial intelligence. Facebook’s new facial recognition technology–called DeepFace–comes astonishingly close to human intelligence in that measure.


Although it’s currently still a research project–which Facebook will present as a paper at a computer vision conference this June–DeepFace has shown itself to be almost as accurate as the human brain when it comes to saying whether two photos show the same person, regardless of whether different lighting or camera angles are used. This has enormous implications for both users and Facebook going forward. Potential applications include everything from improved photo tagging and more accurate online ad selection to more foolproof user authentication. It also poses some serious ethical questions.

So how exactly does it work?

Deconstructing The Technical Problem With Facial Recognition

Although easy for humans, facial recognition is notoriously difficult for computers. Depending on the conditions under which a photo is taken, a computer will struggle enormously to recognize that two images are actually of the same person. Lighting, facial expression, haircuts, and age all contribute to the problem.

Around 2007, facial recognition researchers turned to deep learning neural networks to solve the problem. A neural network is essentially a piece of software designed to simulate how real neurons work. A part of machine learning, deep learning analyzes large amounts of data and develops high-level abstractions by searching for recurring patterns. As part of deep learning, researchers began using photos “from the wild”–meaning everyday photos of normal people rather than images taken under controlled conditions.

At first accuracy fell compared to the results found by earlier researchers. From 99% accuracy using previous, non-deep learning algorithms, accuracy using larger datasets and real-life photographs was suddenly only around the 70% mark. However, it quickly rebounded and moved from 70% to 80%, and then upwards to 90%. Xiaoou Tang–professor in the Department of Information Engineering at the Chinese University of Hong Kong, and one of the world leading experts in facial recognition–describes the use of deep learning neural nets in facial recognition as a “small revolution.”

The Importance Of Training Data

One of the companies using deep neural nets was an Israeli startup called developed a free iOS app called KLIK which learned your friends’ faces and suggested tags for photos instead of users having to manually do this themselves. In 2012 Facebook bought for $100 million. Although the news from the company has been pretty quiet since then, the newly published research paper shows that cofounder Yaniv Taigman and his colleagues have been using their time effectively. While KLIK was a nifty app which got better at recognizing people the more photos you took of them, it also was far from perfect. Pulling a goofy face or changing the lighting meant that mistags could easily happen. With DeepFace’s human-level accuracy, those problems may be a thing of the past.


DeepFace works by using 3-D modeling techniques. Once it has an original image of a face it turns this into a 3-D model, which it can then rotate to generate images of the same face at different angles. Once this is done it can then use its neural network–equipped with 120 million connections–to look for high-level similarities between different photos of the same person.

Aiding the software is the enormous library of images available to the project’s researchers. “Facebook has access to a huge amount of training data,” says Xiaoou Tang. “To put it in context, our algorithm uses 80,000 photos–Facebook has 4 million. One good thing about deep learning is that the more training data you have, the better performance you achieve. This is different to the conventional approach to facial recognition, where once a certain point is reached adding new data will not do much to improve the performance. With deep learning, the more data you have the more the accuracy and performance increases. It’s like the human brain–the longer we’re on this planet, the more we learn and the better we get at solving certain problems.”

A Difficult Tool

Facebook is keen to point out that DeepFace is still a theoretical research project, and not a product being used by Facebook currently. Over time the hope is that that will change, though.

In facial recognition, Facebook has a valuable tool–certainly valuable enough to offset its $100 million investment in “In the immediate future, this technology would most likely be used to improve Facebook’s face-tagging feature, which already uses some face recognition to suggest who might be in each of your photos,” says Neeraj Kumar, a researcher at the University of Washington who has worked on face verification and recognition. “DeepFace would make it more accurate and require less corrections from the user. However, as photos become easier and cheaper to take and all our devices become ‘smarter,’ a critical piece of future software will be personalization and understanding of photos.”

This personalization and understanding can have multiple applications–from targeted adverts based on a deeper understanding of you from your pictures, to the use of facial recognition as a means of creating more secure passwords, along the biometric lines of Apple’s Touch ID. Of course, many of these potential applications raise ethical concerns–which might go some way to explaining why Facebook has been so quiet about the progress made by since its acquisition. Google, by comparison, claimed last year that it won’t add facial recognition features to Google Glass–or approve third-party apps which carry out this function.

“[It’s] a complex subject,” says Seung Yoo, assistant professor of Digital Advertising in the School of Communication at Loyola University Chicago. “The trade is between permission and convenience. In some ways getting permission for Facebook to use our data becomes easier and easier, because they also provide certain benefits to users.” In other words, you get enhanced security features and better recommendations–Facebook gets to use your face.


There is still a way to go for perfecting DeepFace. “There are different ways to measure performance, and this 97% (recognition rate) is for verification,” says Neeraj Kumar, referring to the task of saying whether two faces are the same or not. “The more relevant one for general usage is recognition–given a face, identify who it is, from a database of possibilities.” This latter task is much tougher, although DeepFace has also managed to score highly at it. Its continued success also depends on Facebook users. More users makes the task tougher, but more images per user results in higher accuracy.

Ultimately, however you look at it DeepFace represents a significant advance–and one that is helping (along with other researchers in the field) usher in a new era of facial recognition. “Because the technology has not been accurate enough for most consumer applications so far, I think we–the technology community–haven’t even thought of most of the applications that will arise,” says Kumar.

“With encouraging results like DeepFace and others, I think we’ll soon see an explosion of different uses of this kind of research.”