Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learning
Yi Shen Hanyan Huang Shan Xie
Offline reinforcement learning learns from a static dataset without
interacting with the environment, which ensures security and thus owns a good
prospect of application. However, directly applying naive reinforcement
learning methods usually fails in an offline environment due to function
approximation errors caused by out-of-distribution(OOD) actions. To solve this
problem, existing algorithms mainly penalize the Q-value of OOD actions, the
quality of whose constraints also matter. Imprecise constraints may lead to
suboptimal solutions, while precise constraints require significant
computational costs. In this paper, we propose a novel count-based method for
continuous domains, called Grid-Mapping Pseudo-Count method(GPC), to penalize
the Q-value appropriately and reduce the computational cost. The proposed
method maps the state and action space to discrete space and constrains their
Q-values through the pseudo-count. It is theoretically proved that only a few
conditions are needed to obtain accurate uncertainty constraints in the
proposed method. Moreover, we develop a Grid-Mapping Pseudo-Count Soft
Actor-Critic(GPC-SAC) algorithm using GPC under the Soft Actor-Critic(SAC)
framework to demonstrate the effectiveness of GPC. The experimental results on
D4RL benchmark datasets show that GPC-SAC has better performance and less
computational cost compared to other algorithms.