Boosting Value Decomposition via Unit-Wise Attentive State
  Representation for Cooperative Multi-Agent Reinforcement Learning
Qingpeng Zhao Yuanyang Zhu Zichuan Liu Zhi Wang Chunlin Chen
In cooperative multi-agent reinforcement learning (MARL), the environmental
stochasticity and uncertainties will increase exponentially when the number of
agents increases, which puts hard pressure on how to come up with a compact
latent representation from partial observation for boosting value
decomposition. To tackle these issues, we propose a simple yet powerful method
that alleviates partial observability and efficiently promotes coordination by
introducing the UNit-wise attentive State Representation (UNSR). In UNSR, each
agent learns a compact and disentangled unit-wise state representation
outputted from transformer blocks, and produces its local action-value
function. The proposed UNSR is used to boost the value decomposition with a
multi-head attention mechanism for producing efficient credit assignment in the
mixing network, providing an efficient reasoning path between the individual
value function and joint value function. Experimental results demonstrate that
our method achieves superior performance and data efficiency compared to solid
baselines on the StarCraft II micromanagement challenge. Additional ablation
experiments also help identify the key factors contributing to the performance
of UNSR.