Identify, Estimate and Bound the Uncertainty of Reinforcement Learning
  for Autonomous Driving
Weitao Zhou Zhong Cao Nanshan Deng Kun Jiang Diange Yang
Deep reinforcement learning (DRL) has emerged as a promising approach for
developing more intelligent autonomous vehicles (AVs). A typical DRL
application on AVs is to train a neural network-based driving policy. However,
the black-box nature of neural networks can result in unpredictable decision
failures, making such AVs unreliable. To this end, this work proposes a method
to identify and protect unreliable decisions of a DRL driving policy. The basic
idea is to estimate and constrain the policy's performance uncertainty, which
quantifies potential performance drop due to insufficient training data or
network fitting errors. By constraining the uncertainty, the DRL model's
performance is always greater than that of a baseline policy. The uncertainty
caused by insufficient data is estimated by the bootstrapped method. Then, the
uncertainty caused by the network fitting error is estimated using an ensemble
network. Finally, a baseline policy is added as the performance lower bound to
avoid potential decision failures. The overall framework is called
uncertainty-bound reinforcement learning (UBRL). The proposed UBRL is evaluated
on DRL policies with different amounts of training data, taking an unprotected
left-turn driving case as an example. The result shows that the UBRL method can
identify potentially unreliable decisions of DRL policy. The UBRL guarantees to
outperform baseline policy even when the DRL policy is not well-trained and has
high uncertainty. Meanwhile, the performance of UBRL improves with more
training data. Such a method is valuable for the DRL application on real-road
driving and provides a metric to evaluate a DRL policy.