SS-MAE: Spatial-Spectral Masked Auto-Encoder for Multi-Source Remote
  Sensing Image Classification
Junyan Lin Feng Gao Xiaocheng Shi Junyu Dong Qian Du
Masked image modeling (MIM) is a highly popular and effective self-supervised
learning method for image understanding. Existing MIM-based methods mostly
focus on spatial feature modeling, neglecting spectral feature modeling.
Meanwhile, existing MIM-based methods use Transformer for feature extraction,
some local or high-frequency information may get lost. To this end, we propose
a spatial-spectral masked auto-encoder (SS-MAE) for HSI and LiDAR/SAR data
joint classification. Specifically, SS-MAE consists of a spatial-wise branch
and a spectral-wise branch. The spatial-wise branch masks random patches and
reconstructs missing pixels, while the spectral-wise branch masks random
spectral channels and reconstructs missing channels. Our SS-MAE fully exploits
the spatial and spectral representations of the input data. Furthermore, to
complement local features in the training stage, we add two lightweight CNNs
for feature extraction. Both global and local features are taken into account
for feature modeling. To demonstrate the effectiveness of the proposed SS-MAE,
we conduct extensive experiments on three publicly available datasets.
Extensive experiments on three multi-source datasets verify the superiority of
our SS-MAE compared with several state-of-the-art baselines. The source codes
are available at \url{https://github.com/summitgao/SS-MAE}.