Bag of Policies for Distributional Deep Exploration
Asen Nachkov Luchen Li Giulia Luise Filippo Valdettaro Aldo Faisal
Efficient exploration in complex environments remains a major challenge for
reinforcement learning (RL). Compared to previous Thompson sampling-inspired
mechanisms that enable temporally extended exploration, i.e., deep exploration,
we focus on deep exploration in distributional RL. We develop here a general
purpose approach, Bag of Policies (BoP), that can be built on top of any return
distribution estimator by maintaining a population of its copies. BoP consists
of an ensemble of multiple heads that are updated independently. During
training, each episode is controlled by only one of the heads and the collected
state-action pairs are used to update all heads off-policy, leading to distinct
learning signals for each head which diversify learning and behaviour. To test
whether optimistic ensemble method can improve on distributional RL as did on
scalar RL, by e.g. Bootstrapped DQN, we implement the BoP approach with a
population of distributional actor-critics using Bayesian Distributional Policy
Gradients (BDPG). The population thus approximates a posterior distribution of
return distributions along with a posterior distribution of policies. Another
benefit of building upon BDPG is that it allows to analyze global posterior
uncertainty along with local curiosity bonus simultaneously for exploration. As
BDPG is already an optimistic method, this pairing helps to investigate if
optimism is accumulatable in distributional RL. Overall BoP results in greater
robustness and speed during learning as demonstrated by our experimental
results on ALE Atari games.