PerPLM: Personalized Fine-tuning of Pretrained Language Models via
  Writer-specific Intermediate Learning and Prompts
Daisuke Oba Naoki Yoshinaga Masashi Toyoda
The meanings of words and phrases depend not only on where they are used
(contexts) but also on who use them (writers). Pretrained language models
(PLMs) are powerful tools for capturing context, but they are typically
pretrained and fine-tuned for universal use across different writers. This
study aims to improve the accuracy of text understanding tasks by personalizing
the fine-tuning of PLMs for specific writers. We focus on a general setting
where only the plain text from target writers are available for
personalization. To avoid the cost of fine-tuning and storing multiple copies
of PLMs for different users, we exhaustively explore using writer-specific
prompts to personalize a unified PLM. Since the design and evaluation of these
prompts is an underdeveloped area, we introduce and compare different types of
prompts that are possible in our setting. To maximize the potential of
prompt-based personalized fine-tuning, we propose a personalized intermediate
learning based on masked language modeling to extract task-independent traits
of writers' text. Our experiments, using multiple tasks, datasets, and PLMs,
reveal the nature of different prompts and the effectiveness of our
intermediate learning approach.