Policy Diversity for Cooperative Agents
Mingxi Tan Andong Tian Ludovic Denoyer
Standard cooperative multi-agent reinforcement learning (MARL) methods aim to
find the optimal team cooperative policy to complete a task. However there may
exist multiple different ways of cooperating, which usually are very needed by
domain experts. Therefore, identifying a set of significantly different
policies can alleviate the task complexity for them. Unfortunately, there is a
general lack of effective policy diversity approaches specifically designed for
the multi-agent domain. In this work, we propose a method called
Moment-Matching Policy Diversity to alleviate this problem. This method can
generate different team policies to varying degrees by formalizing the
difference between team policies as the difference in actions of selected
agents in different policies. Theoretically, we show that our method is a
simple way to implement a constrained optimization problem that regularizes the
difference between two trajectory distributions by using the maximum mean
discrepancy. The effectiveness of our approach is demonstrated on a challenging
team-based shooter.