OmniSat: Self-Supervised Modality Fusion for Earth Observation
Guillaume Astruc Nicolas Gonthier Clement Mallet Loic Landrieu
The field of Earth Observations (EO) offers a wealth of data from diverse
sensors, presenting a great opportunity for advancing self-supervised
multimodal learning. However, current multimodal EO datasets and models focus
on a single data type, either mono-date images or time series, which limits
their expressivity. We introduce OmniSat, a novel architecture that exploits
the spatial alignment between multiple EO modalities to learn expressive
multimodal representations without labels. To demonstrate the advantages of
combining modalities of different natures, we augment two existing datasets
with new modalities. As demonstrated on three downstream tasks: forestry, land
cover classification, and crop mapping. OmniSat can learn rich representations
in an unsupervised manner, leading to improved performance in the semi- and
fully-supervised settings, even when only one modality is available for
inference. The code and dataset are available at github.com/gastruc/OmniSat.