Public Release: 

Mind-reading algorithm uses EEG data to reconstruct images based on what we perceive

New technique using EEG shows how our brains perceive faces

University of Toronto


IMAGE: A technique developed by researchers at U of T Scarborough can for the first time reconstruct images of what people perceive based on their brain activity gathered by EEG. view more 

Credit: University of Toronto Scarborough.


The technique developed by Dan Nemrodov, a postdoctoral fellow in Assistant Professor Adrian Nestor's lab at U of T Scarborough, is able to digitally reconstruct images seen by test subjects based on electroencephalography (EEG) data.


For the study, test subjects hooked up to EEG equipment were shown images of faces. Their brain activity was recorded and then used to digitally recreate the image in the subject's mind using a technique based on machine learning algorithms.


And while techniques like fMRI - which measures brain activity by detecting changes in blood flow - can grab finer details of what's going on in specific areas of the brain, EEG has greater practical potential given that it's more common, portable, and inexpensive by comparison. EEG also has greater temporal resolution, meaning it can measure with detail how a percept develops in time right down to milliseconds, explains Nemrodov.

"fMRI captures activity at the time scale of seconds, but EEG captures activity at the millisecond scale. So we can see with very fine detail how the percept of a face develops in our brain using EEG," he says. In fact, the researchers were able to estimate that it takes our brain about 170 milliseconds (0.17 seconds) to form a good representation of a face we see.

This study provides validation that EEG has potential for this type of image reconstruction notes Nemrodov, something many researchers doubted was possible given its apparent limitations. Using EEG data for image reconstruction has great theoretical and practical potential from a neurotechnological standpoint, especially since it's relatively inexpensive and portable.

In terms of next steps, work is currently underway in Nestor's lab to test how image reconstruction based on EEG data could be done using memory and applied to a wider range of objects beyond faces. But it could eventually have wide-ranging clinical applications as well.

"It could provide a means of communication for people who are unable to verbally communicate. Not only could it produce a neural-based reconstruction of what a person is perceiving, but also of what they remember and imagine, of what they want to express," says Nestor.

"It could also have forensic uses for law enforcement in gathering eyewitness information on potential suspects rather than relying on verbal descriptions provided to a sketch artist."

The research, which will be published in the journal eNeuro, was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and by a Connaught New Researcher Award.



