Exploring Lottery Prompts for Pre-trained Language Models
Yulin Chen Ning Ding Xiaobin Wang Shengding Hu Hai-Tao Zheng Zhiyuan Liu Pengjun Xie
Consistently scaling pre-trained language models (PLMs) imposes substantial
burdens on model adaptation, necessitating more efficient alternatives to
conventional fine-tuning. Given the advantage of prompting in the zero-shot
setting and the observed performance fluctuation among different prompts, we
explore the instance-level prompt and their generalizability. By searching
through the prompt space, we first validate the assumption that for every
instance, there is almost always a lottery prompt that induces the correct
prediction from the PLM, and such prompt can be obtained at a low cost thanks
to the inherent ability of PLMs. Meanwhile, we find that some strong lottery
prompts have high performance over the whole training set, and they are
equipped with distinguishable linguistic features. Lastly, we attempt to
generalize the searched strong lottery prompts to unseen data with prompt
ensembling method without any parameter tuning. Experiments are conducted on
various types of NLP classification tasks and demonstrate that the proposed
method can achieve comparable results with other gradient-free and
optimization-free baselines.