Constrained Proximal Policy Optimization
Chengbin Xuan Feng Zhang Faliang Yin Hak-Keung Lam
The problem of constrained reinforcement learning (CRL) holds significant
importance as it provides a framework for addressing critical safety
satisfaction concerns in the field of reinforcement learning (RL). However,
with the introduction of constraint satisfaction, the current CRL methods
necessitate the utilization of second-order optimization or primal-dual
frameworks with additional Lagrangian multipliers, resulting in increased
complexity and inefficiency during implementation. To address these issues, we
propose a novel first-order feasible method named Constrained Proximal Policy
Optimization (CPPO). By treating the CRL problem as a probabilistic inference
problem, our approach integrates the Expectation-Maximization framework to
solve it through two steps: 1) calculating the optimal policy distribution
within the feasible region (E-step), and 2) conducting a first-order update to
adjust the current policy towards the optimal policy obtained in the E-step
(M-step). We establish the relationship between the probability ratios and KL
divergence to convert the E-step into a convex optimization problem.
Furthermore, we develop an iterative heuristic algorithm from a geometric
perspective to solve this problem. Additionally, we introduce a conservative
update mechanism to overcome the constraint violation issue that occurs in the
existing feasible region method. Empirical evaluations conducted in complex and
uncertain environments validate the effectiveness of our proposed method, as it
performs at least as well as other baselines.