Exploration and Comparison of Deep Learning Architectures to Predict
  Brain Response to Realistic Pictures
Riccardo Chimisso Sathya Buršić Paolo Marocco Giuseppe Vizzari Dimitri Ognibene
We present an exploration of machine learning architectures for predicting
brain responses to realistic images on occasion of the Algonauts Challenge
2023. Our research involved extensive experimentation with various pretrained
models. Initially, we employed simpler models to predict brain activity but
gradually introduced more complex architectures utilizing available data and
embeddings generated by large-scale pre-trained models. We encountered typical
difficulties related to machine learning problems, e.g. regularization and
overfitting, as well as issues specific to the challenge, such as difficulty in
combining multiple input encodings, as well as the high dimensionality, unclear
structure, and noisy nature of the output. To overcome these issues we tested
single edge 3D position-based, multi-region of interest (ROI) and hemisphere
predictor models, but we found that employing multiple simple models, each
dedicated to a ROI in each hemisphere of the brain of each subject, yielded the
best results - a single fully connected linear layer with image embeddings
generated by CLIP as input. While we surpassed the challenge baseline, our
results fell short of establishing a robust association with the data.