Value-Aided Conditional Supervised Learning for Offline RL
Jeonghye Kim Suyoung Lee Woojun Kim Youngchul Sung
Offline reinforcement learning (RL) has seen notable advancements through
return-conditioned supervised learning (RCSL) and value-based methods, yet each
approach comes with its own set of practical challenges. Addressing these, we
propose Value-Aided Conditional Supervised Learning (VCS), a method that
effectively synergizes the stability of RCSL with the stitching ability of
value-based methods. Based on the Neural Tangent Kernel analysis to discern
instances where value function may not lead to stable stitching, VCS injects
the value aid into the RCSL's loss function dynamically according to the
trajectory return. Our empirical studies reveal that VCS not only significantly
outperforms both RCSL and value-based methods but also consistently achieves,
or often surpasses, the highest trajectory returns across diverse offline RL
benchmarks. This breakthrough in VCS paves new paths in offline RL, pushing the
limits of what can be achieved and fostering further innovations.