Safe Reinforcement Learning on the Constraint Manifold: Theory and
  Applications
Puze Liu Haitham Bou-Ammar Jan Peters Davide Tateo
Integrating learning-based techniques, especially reinforcement learning,
into robotics is promising for solving complex problems in unstructured
environments. However, most existing approaches are trained in well-tuned
simulators and subsequently deployed on real robots without online fine-tuning.
In this setting, the simulation's realism seriously impacts the deployment's
success rate. Instead, learning with real-world interaction data offers a
promising alternative: not only eliminates the need for a fine-tuned simulator
but also applies to a broader range of tasks where accurate modeling is
unfeasible. One major problem for on-robot reinforcement learning is ensuring
safety, as uncontrolled exploration can cause catastrophic damage to the robot
or the environment. Indeed, safety specifications, often represented as
constraints, can be complex and non-linear, making safety challenging to
guarantee in learning systems. In this paper, we show how we can impose complex
safety constraints on learning-based robotics systems in a principled manner,
both from theoretical and practical points of view. Our approach is based on
the concept of the Constraint Manifold, representing the set of safe robot
configurations. Exploiting differential geometry techniques, i.e., the tangent
space, we can construct a safe action space, allowing learning agents to sample
arbitrary actions while ensuring safety. We demonstrate the method's
effectiveness in a real-world Robot Air Hockey task, showing that our method
can handle high-dimensional tasks with complex constraints. Videos of the real
robot experiments are available on the project website
(https://puzeliu.github.io/TRO-ATACOM).