Distributionally Robust Model-based Reinforcement Learning with Large
  State Spaces
Shyam Sundhar Ramesh Pier Giuseppe Sessa Yifan Hu Andreas Krause Ilija Bogunovic
Three major challenges in reinforcement learning are the complex dynamical
systems with large state spaces, the costly data acquisition processes, and the
deviation of real-world dynamics from the training environment deployment. To
overcome these issues, we study distributionally robust Markov decision
processes with continuous state spaces under the widely used Kullback-Leibler,
chi-square, and total variation uncertainty sets. We propose a model-based
approach that utilizes Gaussian Processes and the maximum variance reduction
algorithm to efficiently learn multi-output nominal transition dynamics,
leveraging access to a generative model (i.e., simulator). We further
demonstrate the statistical sample complexity of the proposed method for
different uncertainty sets. These complexity bounds are independent of the
number of states and extend beyond linear dynamics, ensuring the effectiveness
of our approach in identifying near-optimal distributionally-robust policies.
The proposed method can be further combined with other model-free
distributionally robust reinforcement learning methods to obtain a near-optimal
robust policy. Experimental results demonstrate the robustness of our algorithm
to distributional shifts and its superior performance in terms of the number of
samples needed.