Unsupervised LLM Adaptation for Question Answering
Kuniaki Saito Kihyuk Sohn Chen-Yu Lee Yoshitaka Ushiku
Large language models (LLM) learn diverse knowledge present in the
large-scale training dataset via self-supervised training. Followed by
instruction-tuning, LLM acquires the ability to return correct information for
diverse questions. However, adapting these pre-trained LLMs to new target
domains, such as different organizations or periods, for the question-answering
(QA) task incurs a substantial annotation cost. To tackle this challenge, we
propose a novel task, unsupervised LLM adaptation for question answering. In
this task, we leverage a pre-trained LLM, a publicly available QA dataset
(source data), and unlabeled documents from the target domain. Our goal is to
learn LLM that can answer questions about the target domain. We introduce one
synthetic and two real datasets to evaluate models fine-tuned on the source and
target data, and reveal intriguing insights; (i) fine-tuned models exhibit the
ability to provide correct answers for questions about the target domain even
though they do not see any questions about the information described in the
unlabeled documents, but (ii) they have difficulties in accessing information
located in the middle or at the end of documents, and (iii) this challenge can
be partially mitigated by replacing input tokens with random ones during
adaptation.