PAD: Self-Supervised Pre-Training with Patchwise-Scale Adapter for
  Infrared Images
Tao Zhang Kun Ding Jinyong Wen Yu Xiong Zeyu Zhang Shiming Xiang Chunhong Pan
Self-supervised learning (SSL) for RGB images has achieved significant
success, yet there is still limited research on SSL for infrared images,
primarily due to three prominent challenges: 1) the lack of a suitable
large-scale infrared pre-training dataset, 2) the distinctiveness of non-iconic
infrared images rendering common pre-training tasks like masked image modeling
(MIM) less effective, and 3) the scarcity of fine-grained textures making it
particularly challenging to learn general image features. To address these
issues, we construct a Multi-Scene Infrared Pre-training (MSIP) dataset
comprising 178,756 images, and introduce object-sensitive random RoI cropping,
an image preprocessing method, to tackle the challenge posed by non-iconic
images. To alleviate the impact of weak textures on feature learning, we
propose a pre-training paradigm called Pre-training with ADapter (PAD), which
uses adapters to learn domain-specific features while freezing parameters
pre-trained on ImageNet to retain the general feature extraction capability.
This new paradigm is applicable to any transformer-based SSL method.
Furthermore, to achieve more flexible coordination between pre-trained and
newly-learned features in different layers and patches, a patchwise-scale
adapter with dynamically learnable scale factors is introduced. Extensive
experiments on three downstream tasks show that PAD, with only 1.23M
pre-trainable parameters, outperforms other baseline paradigms including
continual full pre-training on MSIP. Our code and dataset are available at
https://github.com/casiatao/PAD.