Safe, Efficient, Comfort, and Energy-saving Automated Driving through
  Roundabout Based on Deep Reinforcement Learning
Henan Yuan Penghui Li Bart van Arem Liujiang Kang Yongqi Dong
Traffic scenarios in roundabouts pose substantial complexity for automated
driving. Manually mapping all possible scenarios into a state space is
labor-intensive and challenging. Deep reinforcement learning (DRL) with its
ability to learn from interacting with the environment emerges as a promising
solution for training such automated driving models. This study explores,
employs, and implements various DRL algorithms, namely Deep Deterministic
Policy Gradient (DDPG), Proximal Policy Optimization (PPO), and Trust Region
Policy Optimization (TRPO) to instruct automated vehicles' driving through
roundabouts. The driving state space, action space, and reward function are
designed. The reward function considers safety, efficiency, comfort, and energy
consumption to align with real-world requirements. All three tested DRL
algorithms succeed in enabling automated vehicles to drive through the
roundabout. To holistically evaluate the performance of these algorithms, this
study establishes an evaluation methodology considering multiple indicators
such as safety, efficiency, and comfort level. A method employing the Analytic
Hierarchy Process is also developed to weigh these evaluation indicators.
Experimental results on various testing scenarios reveal that the TRPO
algorithm outperforms DDPG and PPO in terms of safety and efficiency, and PPO
performs best in terms of comfort level. Lastly, to verify the model's
adaptability and robustness regarding other driving scenarios, this study also
deploys the model trained by TRPO to a range of different testing scenarios,
e.g., highway driving and merging. Experimental results demonstrate that the
TRPO model trained on only roundabout driving scenarios exhibits a certain
degree of proficiency in highway driving and merging scenarios. This study
provides a foundation for the application of automated driving with DRL in real
traffic environments.