Improving Prompt Tuning with Learned Prompting Layers
Wei Zhu Ming Tan
Prompt tuning prepends a soft prompt to the input embeddings or hidden states
and only optimizes the prompt to adapt pretrained models (PTMs) to downstream
tasks. The previous work manually selects prompt layers which are far from
optimal and failed to exploit the potential of prompt tuning. In this work, we
propose a novel framework, \underline{S}elective \underline{P}rompt
\underline{T}uning (SPT), that learns to select the proper prompt layers by
inserting a prompt controlled by a learnable probabilistic gate at each
intermediate layer. We further propose a novel bi-level optimization framework,
SPT-DARTS, that can better optimize the learnable gates and improve the final
prompt tuning performances of the learned prompt layer settings. We conduct
extensive experiments with ten benchmark datasets under the full-data and
few-shot scenarios. The results demonstrate that our SPT framework can perform
better than the previous state-of-the-art PETuning baselines with comparable or
fewer tunable parameters.