Safety Optimized Reinforcement Learning via Multi-Objective Policy
  Optimization
Homayoun Honari Mehran Ghafarian Tamizi Homayoun Najjaran
Safe reinforcement learning (Safe RL) refers to a class of techniques that
aim to prevent RL algorithms from violating constraints in the process of
decision-making and exploration during trial and error. In this paper, a novel
model-free Safe RL algorithm, formulated based on the multi-objective policy
optimization framework is introduced where the policy is optimized towards
optimality and safety, simultaneously. The optimality is achieved by the
environment reward function that is subsequently shaped using a safety critic.
The advantage of the Safety Optimized RL (SORL) algorithm compared to the
traditional Safe RL algorithms is that it omits the need to constrain the
policy search space. This allows SORL to find a natural tradeoff between safety
and optimality without compromising the performance in terms of either safety
or optimality due to strict search space constraints. Through our theoretical
analysis of SORL, we propose a condition for SORL's converged policy to
guarantee safety and then use it to introduce an aggressiveness parameter that
allows for fine-tuning the mentioned tradeoff. The experimental results
obtained in seven different robotic environments indicate a considerable
reduction in the number of safety violations along with higher, or competitive,
policy returns, in comparison to six different state-of-the-art Safe RL
methods. The results demonstrate the significant superiority of the proposed
SORL algorithm in safety-critical applications.