Composite Active Learning: Towards Multi-Domain Active Learning with
  Theoretical Guarantees
Guang-Yuan Hao Hengguan Huang Haotian Wang Jie Gao Hao Wang
Active learning (AL) aims to improve model performance within a fixed
labeling budget by choosing the most informative data points to label. Existing
AL focuses on the single-domain setting, where all data come from the same
domain (e.g., the same dataset). However, many real-world tasks often involve
multiple domains. For example, in visual recognition, it is often desirable to
train an image classifier that works across different environments (e.g.,
different backgrounds), where images from each environment constitute one
domain. Such a multi-domain AL setting is challenging for prior methods because
they (1) ignore the similarity among different domains when assigning labeling
budget and (2) fail to handle distribution shift of data across different
domains. In this paper, we propose the first general method, dubbed composite
active learning (CAL), for multi-domain AL. Our approach explicitly considers
the domain-level and instance-level information in the problem; CAL first
assigns domain-level budgets according to domain-level importance, which is
estimated by optimizing an upper error bound that we develop; with the
domain-level budgets, CAL then leverages a certain instance-level query
strategy to select samples to label from each domain. Our theoretical analysis
shows that our method achieves a better error bound compared to current AL
methods. Our empirical results demonstrate that our approach significantly
outperforms the state-of-the-art AL methods on both synthetic and real-world
multi-domain datasets. Code is available at
https://github.com/Wang-ML-Lab/multi-domain-active-learning.