Multi-Domain Learning with Modulation Adapters
Ekaterina Iakovleva Karteek Alahari Jakob Verbeek
Deep convolutional networks are ubiquitous in computer vision, due to their
excellent performance across different tasks for various domains. Models are,
however, often trained in isolation for each task, failing to exploit
relatedness between tasks and domains to learn more compact models that
generalise better in low-data regimes. Multi-domain learning aims to handle
related tasks, such as image classification across multiple domains,
simultaneously. Previous work on this problem explored the use of a pre-trained
and fixed domain-agnostic base network, in combination with smaller learnable
domain-specific adaptation modules. In this paper, we introduce Modulation
Adapters, which update the convolutional filter weights of the model in a
multiplicative manner for each task. Parameterising these adaptation weights in
a factored manner allows us to scale the number of per-task parameters in a
flexible manner, and to strike different parameter-accuracy trade-offs. We
evaluate our approach on the Visual Decathlon challenge, composed of ten image
classification tasks across different domains, and on the ImageNet-to-Sketch
benchmark, which consists of six image classification tasks. Our approach
yields excellent results, with accuracies that are comparable to or better than
those of existing state-of-the-art approaches.