Sample Complexity of Offline Distributionally Robust Linear Markov
  Decision Processes
He Wang Laixi Shi Yuejie Chi
In offline reinforcement learning (RL), the absence of active exploration
calls for attention on the model robustness to tackle the sim-to-real gap,
where the discrepancy between the simulated and deployed environments can
significantly undermine the performance of the learned policy. To endow the
learned policy with robustness in a sample-efficient manner in the presence of
high-dimensional state-action space, this paper considers the sample complexity
of distributionally robust linear Markov decision processes (MDPs) with an
uncertainty set characterized by the total variation distance using offline
data. We develop a pessimistic model-based algorithm and establish its sample
complexity bound under minimal data coverage assumptions, which outperforms
prior art by at least $\tilde{O}(d)$, where $d$ is the feature dimension. We
further improve the performance guarantee of the proposed algorithm by
incorporating a carefully-designed variance estimator.