Provable Risk-Sensitive Distributional Reinforcement Learning with
  General Function Approximation
Yu Chen Xiangcheng Zhang Siwei Wang Longbo Huang
In the realm of reinforcement learning (RL), accounting for risk is crucial
for making decisions under uncertainty, particularly in applications where
safety and reliability are paramount. In this paper, we introduce a general
framework on Risk-Sensitive Distributional Reinforcement Learning (RS-DisRL),
with static Lipschitz Risk Measures (LRM) and general function approximation.
Our framework covers a broad class of risk-sensitive RL, and facilitates
analysis of the impact of estimation functions on the effectiveness of RSRL
strategies and evaluation of their sample complexity. We design two innovative
meta-algorithms: \texttt{RS-DisRL-M}, a model-based strategy for model-based
function approximation, and \texttt{RS-DisRL-V}, a model-free approach for
general value function approximation. With our novel estimation techniques via
Least Squares Regression (LSR) and Maximum Likelihood Estimation (MLE) in
distributional RL with augmented Markov Decision Process (MDP), we derive the
first $\widetilde{\mathcal{O}}(\sqrt{K})$ dependency of the regret upper bound
for RSRL with static LRM, marking a pioneering contribution towards
statistically efficient algorithms in this domain.