CaT: Constraints as Terminations for Legged Locomotion Reinforcement
  Learning
Elliot Chane-Sane Pierre-Alexandre Leziart Thomas Flayols Olivier Stasse Philippe Souères Nicolas Mansard
Deep Reinforcement Learning (RL) has demonstrated impressive results in
solving complex robotic tasks such as quadruped locomotion. Yet, current
solvers fail to produce efficient policies respecting hard constraints. In this
work, we advocate for integrating constraints into robot learning and present
Constraints as Terminations (CaT), a novel constrained RL algorithm. Departing
from classical constrained RL formulations, we reformulate constraints through
stochastic terminations during policy learning: any violation of a constraint
triggers a probability of terminating potential future rewards the RL agent
could attain. We propose an algorithmic approach to this formulation, by
minimally modifying widely used off-the-shelf RL algorithms in robot learning
(such as Proximal Policy Optimization). Our approach leads to excellent
constraint adherence without introducing undue complexity and computational
overhead, thus mitigating barriers to broader adoption. Through empirical
evaluation on the real quadruped robot Solo crossing challenging obstacles, we
demonstrate that CaT provides a compelling solution for incorporating
constraints into RL frameworks. Videos and code are available at
https://constraints-as-terminations.github.io.