Adversarially Trained Actor Critic for offline CMDPs
Honghao Wei Xiyue Peng Xin Liu Arnob Ghosh
We propose a Safe Adversarial Trained Actor Critic (SATAC) algorithm for
offline reinforcement learning (RL) with general function approximation in the
presence of limited data coverage. SATAC operates as a two-player Stackelberg
game featuring a refined objective function. The actor (leader player)
optimizes the policy against two adversarially trained value critics (follower
players), who focus on scenarios where the actor's performance is inferior to
the behavior policy. Our framework provides both theoretical guarantees and a
robust deep-RL implementation. Theoretically, we demonstrate that when the
actor employs a no-regret optimization oracle, SATAC achieves two guarantees:
(i) For the first time in the offline RL setting, we establish that SATAC can
produce a policy that outperforms the behavior policy while maintaining the
same level of safety, which is critical to designing an algorithm for offline
RL. (ii) We demonstrate that the algorithm guarantees policy improvement across
a broad range of hyperparameters, indicating its practical robustness.
Additionally, we offer a practical version of SATAC and compare it with
existing state-of-the-art offline safe-RL algorithms in continuous control
environments. SATAC outperforms all baselines across a range of tasks, thus
validating the theoretical performance.