Advantage-Aware Policy Optimization for Offline Reinforcement Learning
Yunpeng Qing Shunyu liu Jingyuan Cong Kaixuan Chen Yihe Zhou Mingli Song
Offline Reinforcement Learning (RL) endeavors to leverage offline datasets to
craft effective agent policy without online interaction, which imposes proper
conservative constraints with the support of behavior policies to tackle the
Out-Of-Distribution (OOD) problem. However, existing works often suffer from
the constraint conflict issue when offline datasets are collected from multiple
behavior policies, i.e., different behavior policies may exhibit inconsistent
actions with distinct returns across the state space. To remedy this issue,
recent Advantage-Weighted (AW) methods prioritize samples with high advantage
values for agent training while inevitably leading to overfitting on these
samples. In this paper, we introduce a novel Advantage-Aware Policy
Optimization (A2PO) method to explicitly construct advantage-aware policy
constraints for offline learning under mixed-quality datasets. Specifically,
A2PO employs a Conditional Variational Auto-Encoder (CVAE) to disentangle the
action distributions of intertwined behavior policies by modeling the advantage
values of all training data as conditional variables. Then the agent can follow
such disentangled action distribution constraints to optimize the
advantage-aware policy towards high advantage values. Extensive experiments
conducted on both the single-quality and mixed-quality datasets of the D4RL
benchmark demonstrate that A2PO yields results superior to state-of-the-art
counterparts. Our code will be made publicly available.