A First-Order Multi-Gradient Algorithm for Multi-Objective Bi-Level
  Optimization
Feiyang Ye Baijiong Lin Xiaofeng Cao Yu Zhang Ivor Tsang
In this paper, we study the Multi-Objective Bi-Level Optimization (MOBLO)
problem, where the upper-level subproblem is a multi-objective optimization
problem and the lower-level subproblem is for scalar optimization. Existing
gradient-based MOBLO algorithms need to compute the Hessian matrix, causing the
computational inefficient problem. To address this, we propose an efficient
first-order multi-gradient method for MOBLO, called FORUM. Specifically, we
reformulate MOBLO problems as a constrained multi-objective optimization (MOO)
problem via the value-function approach. Then we propose a novel multi-gradient
aggregation method to solve the challenging constrained MOO problem.
Theoretically, we provide the complexity analysis to show the efficiency of the
proposed method and a non-asymptotic convergence result. Empirically, extensive
experiments demonstrate the effectiveness and efficiency of the proposed FORUM
method in different learning problems. In particular, it achieves
state-of-the-art performance on three multi-task learning benchmark datasets.