Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image
  Captioning
Cong Yang Zuchao Li Lefei Zhang
Recently, remote sensing image captioning has gained significant attention in
the remote sensing community. Due to the significant differences in spatial
resolution of remote sensing images, existing methods in this field have
predominantly concentrated on the fine-grained extraction of remote sensing
image features, but they cannot effectively handle the semantic consistency
between visual features and textual features. To efficiently align the
image-text, we propose a novel two-stage vision-language pre-training-based
approach to bootstrap interactive image-text alignment for remote sensing image
captioning, called BITA, which relies on the design of a lightweight
interactive Fourier Transformer to better align remote sensing image-text
features. The Fourier layer in the interactive Fourier Transformer is capable
of extracting multi-scale features of remote sensing images in the frequency
domain, thereby reducing the redundancy of remote sensing visual features.
Specifically, the first stage involves preliminary alignment through image-text
contrastive learning, which aligns the learned multi-scale remote sensing
features from the interactive Fourier Transformer with textual features. In the
second stage, the interactive Fourier Transformer connects the frozen image
encoder with a large language model. Then, prefix causal language modeling is
utilized to guide the text generation process using visual features.
Ultimately, across the UCM-caption, RSICD, and NWPU-caption datasets, the
experimental results clearly demonstrate that BITA outperforms other advanced
comparative approaches. The code is available at
https://github.com/yangcong356/BITA.