Scale-Invariant Gradient Aggregation for Constrained Multi-Objective
  Reinforcement Learning
Dohyeong Kim Mineui Hong Jeongho Park Songhwai Oh
Multi-objective reinforcement learning (MORL) aims to find a set of Pareto
optimal policies to cover various preferences. However, to apply MORL in
real-world applications, it is important to find policies that are not only
Pareto optimal but also satisfy pre-defined constraints for safety. To this
end, we propose a constrained MORL (CMORL) algorithm called Constrained
Multi-Objective Gradient Aggregator (CoMOGA). Recognizing the difficulty of
handling multiple objectives and constraints concurrently, CoMOGA relaxes the
original CMORL problem into a constrained optimization problem by transforming
the objectives into additional constraints. This novel transformation process
ensures that the converted constraints are invariant to the objective scales
while having the same effect as the original objectives. We show that the
proposed method converges to a local Pareto optimal policy while satisfying the
predefined constraints. Empirical evaluations across various tasks show that
the proposed method outperforms other baselines by consistently meeting
constraints and demonstrating invariance to the objective scales.