Decomposed Prompt Tuning via Low-Rank Reparameterization
Yao Xiao Lu Xu Jiaxi Li Wei Lu Xiaoli Li
While prompt tuning approaches have achieved competitive performance with
high efficiency, we observe that they invariably employ the same initialization
process, wherein the soft prompt is either randomly initialized or derived from
an existing embedding vocabulary. In contrast to these conventional methods,
this study aims to investigate an alternative way to derive soft prompt. Our
empirical studies show that the soft prompt typically exhibits a low intrinsic
rank characteristic. With such observations, we propose decomposed prompt
tuning, a novel approach that utilizes low-rank matrices to initialize the soft
prompt. Through the low-rank reparameterization, our method significantly
reduces the number of trainable parameters while maintaining effectiveness.
Experimental results on the SuperGLUE benchmark in both high-resource and
low-resource scenarios demonstrate the effectiveness of the proposed method.