PEFTT: Parameter-Efficient Fine-Tuning for low-resource Tibetan
  pre-trained language models
Zhou Mingjun Daiqing Zhuoma Qun Nuo Nyima Tashi
In this era of large language models (LLMs), the traditional training of
models has become increasingly unimaginable for regular users and institutions.
The exploration of efficient fine-tuning for high-resource languages on these
models is an undeniable trend that is gradually gaining popularity. However,
there has been very little exploration for various low-resource languages, such
as Tibetan. Research in Tibetan NLP is inherently scarce and limited. While
there is currently no existing large language model for Tibetan due to its
low-resource nature, that day will undoubtedly arrive. Therefore, research on
efficient fine-tuning for low-resource language models like Tibetan is highly
necessary. Our research can serve as a reference to fill this crucial gap.
Efficient fine-tuning strategies for pre-trained language models (PLMs) in
Tibetan have seen minimal exploration. We conducted three types of efficient
fine-tuning experiments on the publicly available TNCC-title dataset:
"prompt-tuning," "Adapter lightweight fine-tuning," and "prompt-tuning +
Adapter fine-tuning." The experimental results demonstrate significant
improvements using these methods, providing valuable insights for advancing
Tibetan language applications in the context of pre-trained models.