Near-Minimax-Optimal Distributional Reinforcement Learning with a
  Generative Model
Mark Rowland Li Kevin Wenliang Rémi Munos Clare Lyle Yunhao Tang Will Dabney
We propose a new algorithm for model-based distributional reinforcement
learning (RL), and prove that it is minimax-optimal for approximating return
distributions with a generative model (up to logarithmic factors), resolving an
open question of Zhang et al. (2023). Our analysis provides new theoretical
results on categorical approaches to distributional RL, and also introduces a
new distributional Bellman equation, the stochastic categorical CDF Bellman
equation, which we expect to be of independent interest. We also provide an
experimental study comparing several model-based distributional RL algorithms,
with several takeaways for practitioners.