A unified uncertainty-aware exploration: Combining epistemic and
  aleatory uncertainty
Parvin Malekzadeh Ming Hou Konstantinos N. Plataniotis
Exploration is a significant challenge in practical reinforcement learning
(RL), and uncertainty-aware exploration that incorporates the quantification of
epistemic and aleatory uncertainty has been recognized as an effective
exploration strategy. However, capturing the combined effect of aleatory and
epistemic uncertainty for decision-making is difficult. Existing works estimate
aleatory and epistemic uncertainty separately and consider the composite
uncertainty as an additive combination of the two. Nevertheless, the additive
formulation leads to excessive risk-taking behavior, causing instability. In
this paper, we propose an algorithm that clarifies the theoretical connection
between aleatory and epistemic uncertainty, unifies aleatory and epistemic
uncertainty estimation, and quantifies the combined effect of both
uncertainties for a risk-sensitive exploration. Our method builds on a novel
extension of distributional RL that estimates a parameterized return
distribution whose parameters are random variables encoding epistemic
uncertainty. Experimental results on tasks with exploration and risk challenges
show that our method outperforms alternative approaches.