Effectively Prompting Small-sized Language Models for Cross-lingual
  Tasks via Winning Tickets
Mingqi Li Feng Luo
Current soft prompt methods yield limited performance when applied to
small-sized models (fewer than a billion parameters). Deep prompt-tuning, which
entails prepending parameters in each layer for enhanced efficacy, presents a
solution for prompting small-sized models, albeit requiring carefully designed
implementation. In this paper, we introduce the Lottery Ticket Prompt-learning
(LTP) framework that integrates winning tickets with soft prompts. The LTP
offers a simpler implementation and requires only a one-time execution. We
demonstrate LTP on cross-lingual tasks, where prior works rely on external
tools like human-designed multilingual templates and bilingual dictionaries,
which may not be feasible in a low-resource regime. Specifically, we select a
subset of parameters that have been changed the most during the fine-tuning
with the Masked Language Modeling objective. Then, we prepend soft prompts to
the original pre-trained language model and only update the selected parameters
together with prompt-related parameters when adapting to the downstream tasks.
We verify the effectiveness of our LTP framework on cross-lingual tasks,
specifically targeting low-resource languages. Our approach outperforms the
baselines by only updating 20\% of the original parameters.