BLIP-Adapter: Parameter-Efficient Transfer Learning for Mobile
  Screenshot Captioning
Ching-Yu Chiang I-Hua Chang Shih-Wei Liao
This study aims to explore efficient tuning methods for the screenshot
captioning task. Recently, image captioning has seen significant advancements,
but research in captioning tasks for mobile screens remains relatively scarce.
Current datasets and use cases describing user behaviors within product
screenshots are notably limited. Consequently, we sought to fine-tune
pre-existing models for the screenshot captioning task. However, fine-tuning
large pre-trained models can be resource-intensive, requiring considerable
time, computational power, and storage due to the vast number of parameters in
image captioning models. To tackle this challenge, this study proposes a
combination of adapter methods, which necessitates tuning only the additional
modules on the model. These methods are originally designed for vision or
language tasks, and our intention is to apply them to address similar
challenges in screenshot captioning. By freezing the parameters of the image
caption models and training only the weights associated with the methods,
performance comparable to fine-tuning the entire model can be achieved, while
significantly reducing the number of parameters. This study represents the
first comprehensive investigation into the effectiveness of combining adapters
within the context of the screenshot captioning task. Through our experiments
and analyses, this study aims to provide valuable insights into the application
of adapters in vision-language models and contribute to the development of
efficient tuning techniques for the screenshot captioning task. Our study is
available at https://github.com/RainYuGG/BLIP-Adapter