Is Risk-Sensitive Reinforcement Learning Properly Resolved?
Ruiwen Zhou Minghuan Liu Kan Ren Xufang Luo Weinan Zhang Dongsheng Li
Due to the nature of risk management in learning applicable policies,
risk-sensitive reinforcement learning (RSRL) has been realized as an important
direction. RSRL is usually achieved by learning risk-sensitive objectives
characterized by various risk measures, under the framework of distributional
reinforcement learning. However, it remains unclear if the distributional
Bellman operator properly optimizes the RSRL objective in the sense of risk
measures. In this paper, we prove that the existing RSRL methods do not achieve
unbiased optimization and can not guarantee optimality or even improvements
regarding risk measures over accumulated return distributions. To remedy this
issue, we further propose a novel algorithm, namely Trajectory Q-Learning
(TQL), for RSRL problems with provable convergence to the optimal policy. Based
on our new learning architecture, we are free to introduce a general and
practical implementation for different risk measures to learn disparate
risk-sensitive policies. In the experiments, we verify the learnability of our
algorithm and show how our method effectively achieves better performances
toward risk-sensitive objectives.