Uncertainty-aware Distributional Offline Reinforcement Learning
Xiaocong Chen Siyu Wang Tong Yu Lina Yao
Offline reinforcement learning (RL) presents distinct challenges as it relies
solely on observational data. A central concern in this context is ensuring the
safety of the learned policy by quantifying uncertainties associated with
various actions and environmental stochasticity. Traditional approaches
primarily emphasize mitigating epistemic uncertainty by learning risk-averse
policies, often overlooking environmental stochasticity. In this study, we
propose an uncertainty-aware distributional offline RL method to simultaneously
address both epistemic uncertainty and environmental stochasticity. We propose
a model-free offline RL algorithm capable of learning risk-averse policies and
characterizing the entire distribution of discounted cumulative rewards, as
opposed to merely maximizing the expected value of accumulated discounted
returns. Our method is rigorously evaluated through comprehensive experiments
in both risk-sensitive and risk-neutral benchmarks, demonstrating its superior
performance.