StochCA: A Novel Approach for Exploiting Pretrained Models with
  Cross-Attention
Seungwon Seo Suho Lee Sangheum Hwang
Utilizing large-scale pretrained models is a well-known strategy to enhance
performance on various target tasks. It is typically achieved through
fine-tuning pretrained models on target tasks. However, na\"{\i}ve fine-tuning
may not fully leverage knowledge embedded in pretrained models. In this study,
we introduce a novel fine-tuning method, called stochastic cross-attention
(StochCA), specific to Transformer architectures. This method modifies the
Transformer's self-attention mechanism to selectively utilize knowledge from
pretrained models during fine-tuning. Specifically, in each block, instead of
self-attention, cross-attention is performed stochastically according to the
predefined probability, where keys and values are extracted from the
corresponding block of a pretrained model. By doing so, queries and
channel-mixing multi-layer perceptron layers of a target model are fine-tuned
to target tasks to learn how to effectively exploit rich representations of
pretrained models. To verify the effectiveness of StochCA, extensive
experiments are conducted on benchmarks in the areas of transfer learning and
domain generalization, where the exploitation of pretrained models is critical.
Our experimental results show the superiority of StochCA over state-of-the-art
approaches in both areas. Furthermore, we demonstrate that StochCA is
complementary to existing approaches, i.e., it can be combined with them to
further improve performance. Our code is available at
https://github.com/daintlab/stochastic_cross_attention