Multi-modal Instance Refinement for Cross-domain Action Recognition
Yuan Qing Naixing Wu Shaohua Wan Lixin Duan
Unsupervised cross-domain action recognition aims at adapting the model
trained on an existing labeled source domain to a new unlabeled target domain.
Most existing methods solve the task by directly aligning the feature
distributions of source and target domains. However, this would cause negative
transfer during domain adaptation due to some negative training samples in both
domains. In the source domain, some training samples are of low-relevance to
target domain due to the difference in viewpoints, action styles, etc. In the
target domain, there are some ambiguous training samples that can be easily
classified as another type of action under the case of source domain. The
problem of negative transfer has been explored in cross-domain object
detection, while it remains under-explored in cross-domain action recognition.
Therefore, we propose a Multi-modal Instance Refinement (MMIR) method to
alleviate the negative transfer based on reinforcement learning. Specifically,
a reinforcement learning agent is trained in both domains for every modality to
refine the training data by selecting out negative samples from each domain.
Our method finally outperforms several other state-of-the-art baselines in
cross-domain action recognition on the benchmark EPIC-Kitchens dataset, which
demonstrates the advantage of MMIR in reducing negative transfer.