Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model
  Fine-tuning
Zhen-Ru Zhang Chuanqi Tan Haiyang Xu Chengyu Wang Jun Huang Songfang Huang
Fine-tuning large pre-trained language models on various downstream tasks
with whole parameters is prohibitively expensive. Hence, Parameter-efficient
fine-tuning has attracted attention that only optimizes a few task-specific
parameters with the frozen pre-trained model. In this work, we focus on prefix
tuning, which only optimizes continuous prefix vectors (i.e. pseudo tokens)
inserted into Transformer layers. Based on the observation that the learned
syntax and semantics representation varies a lot at different layers, we argue
that the adaptive prefix will be further tailored to each layer than the fixed
one, enabling the fine-tuning more effective and efficient. Thus, we propose
Adaptive Prefix Tuning (APT) to adjust the prefix in terms of both fine-grained
token level and coarse-grained layer level with a gate mechanism. Experiments
on the SuperGLUE and NER datasets show the effectiveness of APT. In addition,
taking the gate as a probing, we validate the efficiency and effectiveness of
the variable prefix.