Surface Masked AutoEncoder: Self-Supervision for Cortical Imaging Data
Simon Dahan Mariana da Silva Daniel Rueckert Emma C Robinson
Self-supervision has been widely explored as a means of addressing the lack
of inductive biases in vision transformer architectures, which limits
generalisation when networks are trained on small datasets. This is crucial in
the context of cortical imaging, where phenotypes are complex and
heterogeneous, but the available datasets are limited in size. This paper
builds upon recent advancements in translating vision transformers to surface
meshes and investigates the potential of Masked AutoEncoder (MAE)
self-supervision for cortical surface learning. By reconstructing surface data
from a masked version of the input, the proposed method effectively models
cortical structure to learn strong representations that translate to improved
performance in downstream tasks. We evaluate our approach on cortical phenotype
regression using the developing Human Connectome Project (dHCP) and demonstrate
that pre-training leads to a 26\% improvement in performance, with an 80\%
faster convergence, compared to models trained from scratch. Furthermore, we
establish that pre-training vision transformer models on large datasets, such
as the UK Biobank (UKB), enables the acquisition of robust representations for
finetuning in low-data scenarios. Our code and pre-trained models are publicly
available at \url{https://github.com/metrics-lab/surface-vision-transformers}.