VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity
  Control
Zi-Yuan Hu Yanyang Li Michael R. Lyu Liwei Wang
As the model size of pre-trained language models (PLMs) grows rapidly, full
fine-tuning becomes prohibitively expensive for model training and storage. In
vision-and-language (VL), parameter-efficient tuning (PET) techniques are
proposed to integrate modular modifications (e.g., Adapter and LoRA) into
encoder-decoder PLMs. By tuning a small set of trainable parameters, these
techniques perform on par with full fine-tuning. However, excessive modular
modifications and neglecting the functionality gap between the encoders and
decoders can lead to performance degradation, while existing PET techniques
(e.g., VL-Adapter) overlook these critical issues. In this paper, we propose a
Vision-and-Language Parameter-Efficient Tuning (VL-PET) framework to impose
effective control over modular modifications via a novel granularity-controlled
mechanism. Considering different granularity-controlled matrices generated by
this mechanism, a variety of model-agnostic VL-PET modules can be instantiated
from our framework for better efficiency and effectiveness trade-offs. We
further propose lightweight PET module designs to enhance VL alignment and
modeling for the encoders and maintain text generation for the decoders.
Extensive experiments conducted on four image-text tasks and four video-text
tasks demonstrate the efficiency, effectiveness and transferability of our
VL-PET framework. In particular, our VL-PET-large with lightweight PET module
designs significantly outperforms VL-Adapter by 2.92% (3.41%) and LoRA by 3.37%
(7.03%) with BART-base (T5-base) on image-text tasks. Furthermore, we validate
the enhanced effect of employing our VL-PET designs on existing PET techniques,
enabling them to achieve significant performance improvements. Our code is
available at https://github.com/HenryHZY/VL-PET.