AstroCLIP: Cross-Modal Pre-Training for Astronomical Foundation Models
Francois Lanusse Liam Parker Siavash Golkar Miles Cranmer Alberto Bietti Michael Eickenberg Geraud Krawezik Michael McCabe Ruben Ohana Mariel Pettee Bruno Regaldo-Saint Blancard Tiberiu Tesileanu Kyunghyun Cho Shirley Ho
We present AstroCLIP, a strategy to facilitate the construction of
astronomical foundation models that bridge the gap between diverse
observational modalities. We demonstrate that a cross-modal contrastive
learning approach between images and optical spectra of galaxies yields highly
informative embeddings of both modalities. In particular, we apply our method
on multi-band images and optical spectra from the Dark Energy Spectroscopic
Instrument (DESI), and show that: (1) these embeddings are well-aligned between
modalities and can be used for accurate cross-modal searches, and (2) these
embeddings encode valuable physical information about the galaxies -- in
particular redshift and stellar mass -- that can be used to achieve competitive
zero- and few- shot predictions without further finetuning. Additionally, in
the process of developing our approach, we also construct a novel,
transformer-based model and pretraining approach for processing galaxy spectra.