Learning Task-preferred Inference Routes for Gradient De-conflict in
  Multi-output DNNs
Yi Sun Xin Xu Jian Li Xiaochang Hu Yifei Shi Ling-Li Zeng
Multi-output deep neural networks(MONs) contain multiple task branches, and
these tasks usually share partial network filters that lead to the entanglement
of different task inference routes. Due to the inconsistent optimization
objectives, the task gradients used for training MONs will interfere with each
other on the shared routes, which will decrease the overall model performance.
To address this issue, we propose a novel gradient de-conflict algorithm named
DR-MGF(Dynamic Routes and Meta-weighted Gradient Fusion) in this work.
Different from existing de-conflict methods, DR-MGF achieves gradient
de-conflict in MONs by learning task-preferred inference routes. The proposed
method is motivated by our experimental findings: the shared filters are not
equally important to different tasks. By designing the learnable task-specific
importance variables, DR-MGF evaluates the importance of filters for different
tasks. Through making the dominances of tasks over filters be proportional to
the task-specific importance of filters, DR-MGF can effectively reduce the
inter-task interference. The task-specific importance variables ultimately
determine task-preferred inference routes at the end of training iterations.
Extensive experimental results on CIFAR, ImageNet, and NYUv2 illustrate that
DR-MGF outperforms the existing de-conflict methods both in prediction accuracy
and convergence speed of MONs. Furthermore, DR-MGF can be extended to general
MONs without modifying the overall network structures.