Domain Generalization Using Large Pretrained Models with
  Mixture-of-Adapters
Gyuseong Lee Wooseok Jang Jin Hyeon Kim Jaewoo Jung Seungryong Kim
Learning a robust vision model despite large distribution shift is essential
for model deployment in real-world settings. Especially, domain generalization
(DG) algorithm aims to maintain the performance of a trained model on different
distributions which were not seen during training. One of the most effective
methods has been leveraging the already learned rich knowledge of large
pretrained models. However, naively fine-tuning large models to DG tasks is
often practically infeasible due to memory limitations, extensive time
requirements for training, and the risk of learned knowledge deterioration.
Recently, parameter-efficient fine-tuning (PEFT) methods have been proposed to
reduce the high computational cost during training and efficiently adapt large
models to downstream tasks. In this work, for the first time, we find that the
use of adapters in PEFT methods not only reduce high computational cost during
training but also serve as an effective regularizer for DG tasks. Surprisingly,
a naive adapter implementation for large models achieve superior performance on
common datasets. However, in situations of large distribution shifts,
additional factors such as optimal amount of regularization due to the strength
of distribution shifts should be considered for a sophisticated adapter
implementation. To address this, we propose a mixture-of-expert based adapter
fine-tuning method, dubbed as mixture-of-adapters (MoA). Specifically, we
employ multiple adapters that have varying capacities, and by using learnable
routers, we allocate each token to a proper adapter. By using both PEFT and MoA
methods, we effectively alleviate the performance deterioration caused by
distribution shifts and achieve state-of-the-art performance on diverse DG
benchmarks.