Adaptive Primal-Dual Method for Safe Reinforcement Learning
Weiqin Chen James Onyejizu Long Vu Lan Hoang Dharmashankar Subramanian Koushik Kar Sandipan Mishra Santiago Paternain
Primal-dual methods have a natural application in Safe Reinforcement Learning
(SRL), posed as a constrained policy optimization problem. In practice however,
applying primal-dual methods to SRL is challenging, due to the inter-dependency
of the learning rate (LR) and Lagrangian multipliers (dual variables) each time
an embedded unconstrained RL problem is solved. In this paper, we propose,
analyze and evaluate adaptive primal-dual (APD) methods for SRL, where two
adaptive LRs are adjusted to the Lagrangian multipliers so as to optimize the
policy in each iteration. We theoretically establish the convergence,
optimality and feasibility of the APD algorithm. Finally, we conduct numerical
evaluation of the practical APD algorithm with four well-known environments in
Bullet-Safey-Gym employing two state-of-the-art SRL algorithms: PPO-Lagrangian
and DDPG-Lagrangian. All experiments show that the practical APD algorithm
outperforms (or achieves comparable performance) and attains more stable
training than the constant LR cases. Additionally, we substantiate the
robustness of selecting the two adaptive LRs by empirical evidence.