Reinforcement Learning by Guided Safe Exploration
Qisong Yang Thiago D. Simão Nils Jansen Simon H. Tindemans Matthijs T. J. Spaan
Safety is critical to broadening the application of reinforcement learning
(RL). Often, we train RL agents in a controlled environment, such as a
laboratory, before deploying them in the real world. However, the real-world
target task might be unknown prior to deployment. Reward-free RL trains an
agent without the reward to adapt quickly once the reward is revealed. We
consider the constrained reward-free setting, where an agent (the guide) learns
to explore safely without the reward signal. This agent is trained in a
controlled environment, which allows unsafe interactions and still provides the
safety signal. After the target task is revealed, safety violations are not
allowed anymore. Thus, the guide is leveraged to compose a safe behaviour
policy. Drawing from transfer learning, we also regularize a target policy (the
student) towards the guide while the student is unreliable and gradually
eliminate the influence of the guide as training progresses. The empirical
analysis shows that this method can achieve safe transfer learning and helps
the student solve the target task faster.