Neural Plasticity-Inspired Foundation Model for Observing the Earth
  Crossing Modalities
Zhitong Xiong Yi Wang Fahong Zhang Adam J. Stewart Joëlle Hanna Damian Borth Ioannis Papoutsis Bertrand Le Saux Gustau Camps-Valls Xiao Xiang Zhu
The development of foundation models has revolutionized our ability to
interpret the Earth's surface using satellite observational data. Traditional
models have been siloed, tailored to specific sensors or data types like
optical, radar, and hyperspectral, each with its own unique characteristics.
This specialization hinders the potential for a holistic analysis that could
benefit from the combined strengths of these diverse data sources. Our novel
approach introduces the Dynamic One-For-All (DOFA) model, leveraging the
concept of neural plasticity in brain science to integrate various data
modalities into a single framework adaptively. This dynamic hypernetwork,
adjusting to different wavelengths, enables a single versatile Transformer
jointly trained on data from five sensors to excel across 12 distinct Earth
observation tasks, including sensors never seen during pretraining. DOFA's
innovative design offers a promising leap towards more accurate, efficient, and
unified Earth observation analysis, showcasing remarkable adaptability and
performance in harnessing the potential of multimodal Earth observation data.