RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question
  Answering
Yuduo Wang Pedram Ghamisi
In recent years, with the rapid advancement of transformer models,
transformer-based multimodal architectures have found wide application in
various downstream tasks, including but not limited to Image Captioning, Visual
Question Answering (VQA), and Image-Text Generation. However, contemporary
approaches to Remote Sensing (RS) VQA often involve resource-intensive
techniques, such as full fine-tuning of large models or the extraction of
image-text features from pre-trained multimodal models, followed by modality
fusion using decoders. These approaches demand significant computational
resources and time, and a considerable number of trainable parameters are
introduced. To address these challenges, we introduce a novel method known as
RSAdapter, which prioritizes runtime and parameter efficiency. RSAdapter
comprises two key components: the Parallel Adapter and an additional linear
transformation layer inserted after each fully connected (FC) layer within the
Adapter. This approach not only improves adaptation to pre-trained multimodal
models but also allows the parameters of the linear transformation layer to be
integrated into the preceding FC layers during inference, reducing inference
costs. To demonstrate the effectiveness of RSAdapter, we conduct an extensive
series of experiments using three distinct RS-VQA datasets and achieve
state-of-the-art results on all three datasets. The code for RSAdapter will be
available online at https://github.com/Y-D-Wang/RSAdapter.