Diverse Projection Ensembles for Distributional Reinforcement Learning
Moritz A. Zanger Wendelin Böhmer Matthijs T. J. Spaan
In contrast to classical reinforcement learning, distributional reinforcement
learning algorithms aim to learn the distribution of returns rather than their
expected value. Since the nature of the return distribution is generally
unknown a priori or arbitrarily complex, a common approach finds approximations
within a set of representable, parametric distributions. Typically, this
involves a projection of the unconstrained distribution onto the set of
simplified distributions. We argue that this projection step entails a strong
inductive bias when coupled with neural networks and gradient descent, thereby
profoundly impacting the generalization behavior of learned models. In order to
facilitate reliable uncertainty estimation through diversity, this work studies
the combination of several different projections and representations in a
distributional ensemble. We establish theoretical properties of such projection
ensembles and derive an algorithm that uses ensemble disagreement, measured by
the average $1$-Wasserstein distance, as a bonus for deep exploration. We
evaluate our algorithm on the behavior suite benchmark and find that diverse
projection ensembles lead to significant performance improvements over existing
methods on a wide variety of tasks with the most pronounced gains in directed
exploration problems.