Offline Goal-Conditioned Reinforcement Learning for Safety-Critical
  Tasks with Recovery Policy
Chenyang Cao Zichen Yan Renhao Lu Junbo Tan Xueqian Wang
Offline goal-conditioned reinforcement learning (GCRL) aims at solving
goal-reaching tasks with sparse rewards from an offline dataset. While prior
work has demonstrated various approaches for agents to learn near-optimal
policies, these methods encounter limitations when dealing with diverse
constraints in complex environments, such as safety constraints. Some of these
approaches prioritize goal attainment without considering safety, while others
excessively focus on safety at the expense of training efficiency. In this
paper, we study the problem of constrained offline GCRL and propose a new
method called Recovery-based Supervised Learning (RbSL) to accomplish
safety-critical tasks with various goals. To evaluate the method performance,
we build a benchmark based on the robot-fetching environment with a randomly
positioned obstacle and use expert or random policies to generate an offline
dataset. We compare RbSL with three offline GCRL algorithms and one offline
safe RL algorithm. As a result, our method outperforms the existing
state-of-the-art methods to a large extent. Furthermore, we validate the
practicality and effectiveness of RbSL by deploying it on a real Panda
manipulator. Code is available at https://github.com/Sunlighted/RbSL.git.