Decoding visual brain representations from electroencephalography
  through Knowledge Distillation and latent diffusion models
Matteo Ferrante Tommaso Boccato Stefano Bargione Nicola Toschi
Decoding visual representations from human brain activity has emerged as a
thriving research domain, particularly in the context of brain-computer
interfaces. Our study presents an innovative method that employs to classify
and reconstruct images from the ImageNet dataset using electroencephalography
(EEG) data from subjects that had viewed the images themselves (i.e. "brain
decoding"). We analyzed EEG recordings from 6 participants, each exposed to 50
images spanning 40 unique semantic categories. These EEG readings were
converted into spectrograms, which were then used to train a convolutional
neural network (CNN), integrated with a knowledge distillation procedure based
on a pre-trained Contrastive Language-Image Pre-Training (CLIP)-based image
classification teacher network. This strategy allowed our model to attain a
top-5 accuracy of 80%, significantly outperforming a standard CNN and various
RNN-based benchmarks. Additionally, we incorporated an image reconstruction
mechanism based on pre-trained latent diffusion models, which allowed us to
generate an estimate of the images which had elicited EEG activity. Therefore,
our architecture not only decodes images from neural activity but also offers a
credible image reconstruction from EEG only, paving the way for e.g. swift,
individualized feedback experiments. Our research represents a significant step
forward in connecting neural signals with visual cognition.