Long and Short-Term Constraints Driven Safe Reinforcement Learning for
  Autonomous Driving
Xuemin Hu Pan Chen Yijun Wen Bo Tang Long Chen
Reinforcement learning (RL) has been widely used in decision-making tasks,
but it cannot guarantee the agent's safety in the training process due to the
requirements of interaction with the environment, which seriously limits its
industrial applications such as autonomous driving. Safe RL methods are
developed to handle this issue by constraining the expected safety violation
costs as a training objective, but they still permit unsafe state occurrence,
which is unacceptable in autonomous driving tasks. Moreover, these methods are
difficult to achieve a balance between the cost and return expectations, which
leads to learning performance degradation for the algorithms. In this paper, we
propose a novel algorithm based on the long and short-term constraints (LSTC)
for safe RL. The short-term constraint aims to guarantee the short-term state
safety that the vehicle explores, while the long-term constraint ensures the
overall safety of the vehicle throughout the decision-making process. In
addition, we develop a safe RL method with dual-constraint optimization based
on the Lagrange multiplier to optimize the training process for end-to-end
autonomous driving. Comprehensive experiments were conducted on the MetaDrive
simulator. Experimental results demonstrate that the proposed method achieves
higher safety in continuous state and action tasks, and exhibits higher
exploration performance in long-distance decision-making tasks compared with
state-of-the-art methods.