COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
Hongxin Zhang Zeyuan Wang Qiushi Lyu Zheyuan Zhang Sunli Chen Tianmin Shu Yilun Du Chuang Gan
In this paper, we investigate the problem of embodied multi-agent
cooperation, where decentralized agents must cooperate given only partial
egocentric views of the world. To effectively plan in this setting, in contrast
to learning world dynamics in a single-agent scenario, we must simulate world
dynamics conditioned on an arbitrary number of agents' actions given only
partial egocentric visual observations of the world. To address this issue of
partial observability, we first train generative models to estimate the overall
world state given partial egocentric observations. To enable accurate
simulation of multiple sets of actions on this world state, we then propose to
learn a compositional world model for multi-agent cooperation by factorizing
the naturally composable joint actions of multiple agents and compositionally
generating the video. By leveraging this compositional world model, in
combination with Vision Language Models to infer the actions of other agents,
we can use a tree search procedure to integrate these modules and facilitate
online cooperative planning. To evaluate the efficacy of our methods, we create
two challenging embodied multi-agent long-horizon cooperation tasks using the
ThreeDWorld simulator and conduct experiments with 2-4 agents. The results show
our compositional world model is effective and the framework enables the
embodied agents to cooperate efficiently with different agents across various
tasks and an arbitrary number of agents, showing the promising future of our
proposed framework. More videos can be found at
https://vis-www.cs.umass.edu/combo/.