Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to
  Pre-trained Language Models Memories
Shizhe Diao Tianyang Xu Ruijia Xu Jiawei Wang Tong Zhang
Pre-trained language models (PLMs) demonstrate excellent abilities to
understand texts in the generic domain while struggling in a specific domain.
Although continued pre-training on a large domain-specific corpus is effective,
it is costly to tune all the parameters on the domain. In this paper, we
investigate whether we can adapt PLMs both effectively and efficiently by only
tuning a few parameters. Specifically, we decouple the feed-forward networks
(FFNs) of the Transformer architecture into two parts: the original pre-trained
FFNs to maintain the old-domain knowledge and our novel domain-specific
adapters to inject domain-specific knowledge in parallel. Then we adopt a
mixture-of-adapters gate to fuse the knowledge from different domain adapters
dynamically. Our proposed Mixture-of-Domain-Adapters (MixDA) employs a
two-stage adapter-tuning strategy that leverages both unlabeled data and
labeled data to help the domain adaptation: i) domain-specific adapter on
unlabeled data; followed by ii) the task-specific adapter on labeled data.
MixDA can be seamlessly plugged into the pretraining-finetuning paradigm and
our experiments demonstrate that MixDA achieves superior performance on
in-domain tasks (GLUE), out-of-domain tasks (ChemProt, RCT, IMDB, Amazon), and
knowledge-intensive tasks (KILT). Further analyses demonstrate the reliability,
scalability, and efficiency of our method. The code is available at
https://github.com/Amano-Aki/Mixture-of-Domain-Adapters.