TCM-GPT: Efficient Pre-training of Large Language Models for Domain
  Adaptation in Traditional Chinese Medicine
Guoxing Yang Jianyu Shi Zan Wang Xiaohong Liu Guangyu Wang
Pre-training and fine-tuning have emerged as a promising paradigm across
various natural language processing (NLP) tasks. The effectiveness of
pretrained large language models (LLM) has witnessed further enhancement,
holding potential for applications in the field of medicine, particularly in
the context of Traditional Chinese Medicine (TCM). However, the application of
these general models to specific domains often yields suboptimal results,
primarily due to challenges like lack of domain knowledge, unique objectives,
and computational efficiency. Furthermore, their effectiveness in specialized
domains, such as Traditional Chinese Medicine, requires comprehensive
evaluation. To address the above issues, we propose a novel domain specific
TCMDA (TCM Domain Adaptation) approach, efficient pre-training with
domain-specific corpus. Specifically, we first construct a large TCM-specific
corpus, TCM-Corpus-1B, by identifying domain keywords and retreving from
general corpus. Then, our TCMDA leverages the LoRA which freezes the pretrained
model's weights and uses rank decomposition matrices to efficiently train
specific dense layers for pre-training and fine-tuning, efficiently aligning
the model with TCM-related tasks, namely TCM-GPT-7B. We further conducted
extensive experiments on two TCM tasks, including TCM examination and TCM
diagnosis. TCM-GPT-7B archived the best performance across both datasets,
outperforming other models by relative increments of 17% and 12% in accuracy,
respectively. To the best of our knowledge, our study represents the pioneering
validation of domain adaptation of a large language model with 7 billion
parameters in TCM domain. We will release both TCMCorpus-1B and TCM-GPT-7B
model once accepted to facilitate interdisciplinary development in TCM and NLP,
serving as the foundation for further study.