Constrained Reinforcement Learning with Smoothed Log Barrier Function
Baohe Zhang Yuan Zhang Lilli Frison Thomas Brox Joschka Bödecker
Reinforcement Learning (RL) has been widely applied to many control tasks and
substantially improved the performances compared to conventional control
methods in many domains where the reward function is well defined. However, for
many real-world problems, it is often more convenient to formulate optimization
problems in terms of rewards and constraints simultaneously. Optimizing such
constrained problems via reward shaping can be difficult as it requires tedious
manual tuning of reward functions with several interacting terms. Recent
formulations which include constraints mostly require a pre-training phase,
which often needs human expertise to collect data or assumes having a
sub-optimal policy readily available. We propose a new constrained RL method
called CSAC-LB (Constrained Soft Actor-Critic with Log Barrier Function), which
achieves competitive performance without any pre-training by applying a linear
smoothed log barrier function to an additional safety critic. It implements an
adaptive penalty for policy learning and alleviates the numerical issues that
are known to complicate the application of the log barrier function method. As
a result, we show that with CSAC-LB, we achieve state-of-the-art performance on
several constrained control tasks with different levels of difficulty and
evaluate our methods in a locomotion task on a real quadruped robot platform.