Rethinking Transformers Pre-training for Multi-Spectral Satellite
  Imagery
Mubashir Noman Muzammal Naseer Hisham Cholakkal Rao Muhammad Anwar Salman Khan Fahad Shahbaz Khan
Recent advances in unsupervised learning have demonstrated the ability of
large vision models to achieve promising results on downstream tasks by
pre-training on large amount of unlabelled data. Such pre-training techniques
have also been explored recently in the remote sensing domain due to the
availability of large amount of unlabelled data. Different from standard
natural image datasets, remote sensing data is acquired from various sensor
technologies and exhibit diverse range of scale variations as well as
modalities. Existing satellite image pre-training methods either ignore the
scale information present in the remote sensing imagery or restrict themselves
to use only a single type of data modality. In this paper, we re-visit
transformers pre-training and leverage multi-scale information that is
effectively utilized with multiple modalities. Our proposed approach, named
SatMAE++, performs multi-scale pre-training and utilizes convolution based
upsampling blocks to reconstruct the image at higher scales making it
extensible to include more scales. Compared to existing works, the proposed
SatMAE++ with multi-scale pre-training is equally effective for both optical as
well as multi-spectral imagery. Extensive experiments on six datasets reveal
the merits of proposed contributions, leading to state-of-the-art performance
on all datasets. SatMAE++ achieves mean average precision (mAP) gain of 2.5\%
for multi-label classification task on BigEarthNet dataset. Our code and
pre-trained models are available at \url{https://github.com/techmn/satmae_pp}.