Res-Attn : An Enhanced Res-Tuning Approach with Lightweight Attention
  Mechanism
Chaojie Mao Zeyinzi Jiang
Res-Tuning introduces a flexible and efficient paradigm for model tuning,
showing that tuners decoupled from the backbone network can achieve performance
comparable to traditional methods. Existing methods commonly construct the
tuner as a set of trainable low-rank decomposition matrices, positing that a
low-rank subspace suffices for adapting pre-trained foundational models to new
scenarios. In this work, we present an advanced, efficient tuner augmented with
low-rank attention, termed Res-Attn , which also adheres to the Res-Tuning
framework. Res-Attn utilizes a parallel multi-head attention module equipped
with low-rank projections for query, key, and value to execute streamlined
attention operations. Through training this lightweight attention module,
Res-Attn facilitates adaptation to new scenarios. Our extensive experiments
across a range of discriminative and generative tasks showcase the superior
performance of our method when compared to existing alternatives