Safe Reinforcement Learning with Dead-Ends Avoidance and Recovery
Xiao Zhang Hai Zhang Hongtu Zhou Chang Huang Di Zhang Chen Ye Junqiao Zhao
Safety is one of the main challenges in applying reinforcement learning to
realistic environmental tasks. To ensure safety during and after training
process, existing methods tend to adopt overly conservative policy to avoid
unsafe situations. However, overly conservative policy severely hinders the
exploration, and makes the algorithms substantially less rewarding. In this
paper, we propose a method to construct a boundary that discriminates safe and
unsafe states. The boundary we construct is equivalent to distinguishing
dead-end states, indicating the maximum extent to which safe exploration is
guaranteed, and thus has minimum limitation on exploration. Similar to Recovery
Reinforcement Learning, we utilize a decoupled RL framework to learn two
policies, (1) a task policy that only considers improving the task performance,
and (2) a recovery policy that maximizes safety. The recovery policy and a
corresponding safety critic are pretrained on an offline dataset, in which the
safety critic evaluates upper bound of safety in each state as awareness of
environmental safety for the agent. During online training, a behavior
correction mechanism is adopted, ensuring the agent to interact with the
environment using safe actions only. Finally, experiments of continuous control
tasks demonstrate that our approach has better task performance with less
safety violations than state-of-the-art algorithms.