The machine learning and computer vision community is witnessing an unprecedented rate of new tasks being proposed and addressed, thanks to the power of deep convolutional networks to find complex mappings from X to Y. The advent of each task often accompanies the release of a large-scale human-labeled dataset, for supervised training of the deep network. However, it is expensive and time-consuming to manually label sufficient amount of training data. Therefore, it is important to develop algorithms that can leverage off-the-shelf labeled dataset to learn useful knowledge for the target task. While previous works mostly focus on transfer learning from a single source, we study multi-source transfer across domains and tasks (MS-DTT), in a semi-supervised setting. We propose GradMix, a model-agnostic method applicable to any model trained with gradient-based learning rule. GradMix transfers knowledge via gradient descent, by weighting and mixing the gradients from all sources during training. Our method follows a meta-learning objective, by assigning layer-wise weights to the source gradients, such that the combined gradient follows the direction that can minimize the loss for a small set of samples from the target dataset. In addition, we propose to adaptively adjust the learning rate for each mini-batch based on its importance to the target task, and a pseudo-labeling method to leverage the unlabeled samples in the target domain. We perform experiments on two MS-DTT tasks: digit recognition and action recognition, and demonstrate the advantageous performance of the proposed method against multiple baselines. Deep convolutional networks (ConvNets) have significantly improved the state-of-the-art for visual recognition, by finding complex mappings from X to Y. Unfortunately, these impressive gains in performance come only when massive amounts of paired labeled data (x, y) s.t. x ∈ X , y ∈ Y are available for supervised training. For many application domains, it is often prohibitive to manually label sufficient training data, due to the significant amount of human efforts involved. Hence, there is strong incentive to develop algorithms that can reduce the burden of manual labeling, typically by leveraging off-the-shelf labeled datasets from other related domains and tasks.There has been a large amount of efforts in the research community to address adapting deep models across domains BID5 BID16 BID31 , to transfer knowledge across tasks BID17 BID7 BID34 , and to learn efficiently in a few shot manner BID4 BID22 BID23 . However, most works focus on a single-source and single-target scenario. Recently, some works BID33 BID19 propose deep approaches for multi-source domain adaptation, but they assume that the source and target domains have shared label space (task).In many computer vision applications, there often exist multiple labeled datasets available from different domains and/or tasks related to the target application. Hence , it is important and practically valuable that we can transfer knowledge from as many source datasets as possible. In this work, we formalize this problem as multi-source domain and task transfer (MS-DTT). Given a set of labeled source dataset, S = {S 1 , S 2 , ..., S k }, we aim to transfer knowledge to a sparsely labeled target dataset T . Each source dataset S i could come from a different domain compared to T , or from a different task, or different in both domain and task. We focus on a semi-supervised setting, where only few samples in T have labels.Most works achieve domain transfer by aligning the feature distribution of source domain and target domain BID15 BID5 BID30 BID19 BID33 . However, this method could be suboptimal for MS-DTT. The reason is that in MS-DTT, the distribution of source data p(x Si , y Si ) and target data p(x T , y T ) could be significantly different in both input space and label space, thus simply aligning their input space may generate indiscriminative features for the target classes. In addition, feature alignment introduces additional layers and loss terms, which require careful design to perform well.In this work, we propose a generic and scalable method, namely GradMix, for semi-supervised MS-DTT. GradMix is a model-agnostic method, applicable to any model that uses gradient-based learning rule. Our method does not introduce extra layers or loss functions for feature alignment. Instead, we perform knowledge transfer via gradient descent, by weighting and mixing the gradients from all the source datasets during training. We follow a meta-learning paradigm and model the most basic assumption: the combined gradient should minimize the loss for a set of unbiased samples from the target dataset. We propose an online method to weight and mix the source gradients at each training iteration, such that the knowledge most useful for the target task is preserved through the gradient update. Our method can adaptively adjust the learning rate for each mini-batch based on its importance to the target task. In addition, we propose a pseudo-labeling method based on model ensemble to learn from the unlabeled data in target domain. We perform extensive experiments on two sets of MS-DTT task, including digit recognition and action recognition, and demonstrate the advantageous performance of the proposed method compared to multiple baselines. Our code is available at https://www.url.com . In this work, we propose GradMix, a method for semi-supervised MS-DTT: multi-source domain and task transfer. GradMix assigns layer-wise weights to the gradients calculated from each source objective, in a way such that the combined gradient can optimize the target objective, measured by the loss on a small validation set. GradMix can adaptively adjust the learning rate for each mini-batch based on its importance to the target task. In addition, we assign pseudo-labels to the unlabeled samples using model ensembles, and consider the pseudo-labeled dataset as a source during training. We validate the effectiveness our method with extensive experiments on two MS-DTT settings, namely digit recognition and action recognition. GradMix is a generic framework applicable to any models trained with gradient descent. For future work, we intend to extend GradMix to other problems where labeled data for the target task is expensive to acquire, such as image captioning.
We propose a gradient-based method to transfer knowledge from multiple sources across different domains and tasks. This paper proposes to combine the gradients of source domains to help the learning in the target domain. 