Mediated Multi-Agent Reinforcement Learning
Dmitry Ivanov Ilya Zisman Kirill Chernyshev
The majority of Multi-Agent Reinforcement Learning (MARL) literature equates
the cooperation of self-interested agents in mixed environments to the problem
of social welfare maximization, allowing agents to arbitrarily share rewards
and private information. This results in agents that forgo their individual
goals in favour of social good, which can potentially be exploited by selfish
defectors. We argue that cooperation also requires agents' identities and
boundaries to be respected by making sure that the emergent behaviour is an
equilibrium, i.e., a convention that no agent can deviate from and receive
higher individual payoffs. Inspired by advances in mechanism design, we propose
to solve the problem of cooperation, defined as finding socially beneficial
equilibrium, by using mediators. A mediator is a benevolent entity that may act
on behalf of agents, but only for the agents that agree to it. We show how a
mediator can be trained alongside agents with policy gradient to maximize
social welfare subject to constraints that encourage agents to cooperate
through the mediator. Our experiments in matrix and iterative games highlight
the potential power of applying mediators in MARL.