Is Prompt-Based Finetuning Always Better than Vanilla Finetuning?
  Insights from Cross-Lingual Language Understanding
Bolei Ma Ercong Nie Helmut Schmid Hinrich Schütze
Multilingual pretrained language models (MPLMs) have demonstrated substantial
performance improvements in zero-shot cross-lingual transfer across various
natural language understanding tasks by finetuning MPLMs on task-specific
labelled data of a source language (e.g. English) and evaluating on a wide
range of target languages. Recent studies show that prompt-based finetuning
surpasses regular finetuning in few-shot scenarios. However, the exploration of
prompt-based learning in multilingual tasks remains limited. In this study, we
propose the ProFiT pipeline to investigate the cross-lingual capabilities of
Prompt-based Finetuning. We conduct comprehensive experiments on diverse
cross-lingual language understanding tasks (sentiment classification,
paraphrase identification, and natural language inference) and empirically
analyze the variation trends of prompt-based finetuning performance in
cross-lingual transfer across different few-shot and full-data settings. Our
results reveal the effectiveness and versatility of prompt-based finetuning in
cross-lingual language understanding. Our findings indicate that prompt-based
finetuning outperforms vanilla finetuning in full-data scenarios and exhibits
greater advantages in few-shot scenarios, with different performance patterns
dependent on task types. Additionally, we analyze underlying factors such as
language similarity and pretraining data size that impact the cross-lingual
performance of prompt-based finetuning. Overall, our work provides valuable
insights into the cross-lingual prowess of prompt-based finetuning.