Reaching Consensus in Cooperative Multi-Agent Reinforcement Learning
  with Goal Imagination
Liangzhou Wang Kaiwen Zhu Fengming Zhu Xinghu Yao Shujie Zhang Deheng Ye Haobo Fu Qiang Fu Wei Yang
Reaching consensus is key to multi-agent coordination. To accomplish a
cooperative task, agents need to coherently select optimal joint actions to
maximize the team reward. However, current cooperative multi-agent
reinforcement learning (MARL) methods usually do not explicitly take consensus
into consideration, which may cause miscoordination problem. In this paper, we
propose a model-based consensus mechanism to explicitly coordinate multiple
agents. The proposed Multi-agent Goal Imagination (MAGI) framework guides
agents to reach consensus with an Imagined common goal. The common goal is an
achievable state with high value, which is obtained by sampling from the
distribution of future states. We directly model this distribution with a
self-supervised generative model, thus alleviating the "curse of dimensinality"
problem induced by multi-agent multi-step policy rollout commonly used in
model-based methods. We show that such efficient consensus mechanism can guide
all agents cooperatively reaching valuable future states. Results on
Multi-agent Particle-Environments and Google Research Football environment
demonstrate the superiority of MAGI in both sample efficiency and performance.