DEPHN: Different Expression Parallel Heterogeneous Network using virtual
  gradient optimization for Multi-task Learning
Menglin Kong Ri Su Shaojie Zhao Muzhou Hou
Recommendation system algorithm based on multi-task learning (MTL) is the
major method for Internet operators to understand users and predict their
behaviors in the multi-behavior scenario of platform. Task correlation is an
important consideration of MTL goals, traditional models use shared-bottom
models and gating experts to realize shared representation learning and
information differentiation. However, The relationship between real-world tasks
is often more complex than existing methods do not handle properly sharing
information. In this paper, we propose an Different Expression Parallel
Heterogeneous Network (DEPHN) to model multiple tasks simultaneously. DEPHN
constructs the experts at the bottom of the model by using different feature
interaction methods to improve the generalization ability of the shared
information flow. In view of the model's differentiating ability for different
task information flows, DEPHN uses feature explicit mapping and virtual
gradient coefficient for expert gating during the training process, and
adaptively adjusts the learning intensity of the gated unit by considering the
difference of gating values and task correlation. Extensive experiments on
artificial and real-world datasets demonstrate that our proposed method can
capture task correlation in complex situations and achieve better performance
than baseline models\footnote{Accepted in IJCNN2023}.