

But those tests were performed on a dataset with only 13,000 images — fewer people than attend an average professional U.S. soccer game. What happens to their performance as those crowds grow to the size of a major U.S. city?




Google’s FaceNet showed the strongest performance on one test, dropping from near-perfect accuracy when confronted with a smaller number of images to 75 percent on the million person test. A team from Russia’s N-TechLab came out on top on another test set, dropping to 73 percent.

By contrast, the accuracy rates of other algorithms that had performed well at a small scale dropped by much larger percentages to as low as 33 percent accuracy when confronted with the harder task.


The MegaFace challenge tested the algorithms on verification, or how well they could correctly identify whether two photos were of the same person. That’s how an iPhone security feature, for instance, could recognize your face and decide whether to unlock your phone instead of asking you to type in a password.

“What happens if you lose your phone in a train station in Amsterdam and someone tries to steal it?” said Kemelmacher-Shlizerman, who co-leads the UW Graphics and Imaging Laboratory (GRAIL.) “I’d want certainty that my phone can correctly identify me out of a million people — or 7 billion — not just 10,000 or so.”

They also tested the algorithms on identification, or how accurately they could find a match to the photo of a single individual to a different photo of the same person buried among a million “distractors.” That’s what happens, for instance, when law enforcement have a single photograph of a criminal suspect and are combing through images taken on a subway platform or airport to see if the person is trying to escape.

“You can see where the hard problems are — recognizing people across different ages is an unsolved problem. So is identifying people from their doppelgängers  and matching people who are in varying poses like side views to frontal views,” said Kemelmacher-Shlizerman. The paper also analyses age and pose invariance in face recognition when evaluated at scale.


The MegaFace challenge is ongoing and still accepting results.

The team’s next steps include assembling a half a million identities — each with a number of photographs — for a dataset that will be used to train facial recognition algorithms. This will help level the playing field and test which algorithms outperform others given the same amount of large scale training data, as most researchers don’t have access to image collections as large as Google’s or Facebook’s. The training set will be released towards the end of the summer.

“State-of-the-art deep neural network algorithms have millions of parameters to learn and require a plethora of examples to accurately tune them,” said Aaron Nech, a UW computer science and engineering master’s student working on the training dataset. “Unlike people, these models are initially a blank slate. Having diversity in the data, such as the intricate identity cues found across more than 500,000 unique individuals, can increase algorithm performance by providing examples of situations not yet seen.”

