Subequivariant Graph Reinforcement Learning in 3D Environments
Runfa Chen Jiaqi Han Fuchun Sun Wenbing Huang
Learning a shared policy that guides the locomotion of different agents is of
core interest in Reinforcement Learning (RL), which leads to the study of
morphology-agnostic RL. However, existing benchmarks are highly restrictive in
the choice of starting point and target point, constraining the movement of the
agents within 2D space. In this work, we propose a novel setup for
morphology-agnostic RL, dubbed Subequivariant Graph RL in 3D environments
(3D-SGRL). Specifically, we first introduce a new set of more practical yet
challenging benchmarks in 3D space that allows the agent to have full
Degree-of-Freedoms to explore in arbitrary directions starting from arbitrary
configurations. Moreover, to optimize the policy over the enlarged state-action
space, we propose to inject geometric symmetry, i.e., subequivariance, into the
modeling of the policy and Q-function such that the policy can generalize to
all directions, improving exploration efficiency. This goal is achieved by a
novel SubEquivariant Transformer (SET) that permits expressive message
exchange. Finally, we evaluate the proposed method on the proposed benchmarks,
where our method consistently and significantly outperforms existing approaches
on single-task, multi-task, and zero-shot generalization scenarios. Extensive
ablations are also conducted to verify our design. Code and videos are
available on our project page: https://alpc91.github.io/SGRL/.