On Imitation in Mean-field Games
Giorgia Ramponi Pavel Kolev Olivier Pietquin Niao He Mathieu Laurière Matthieu Geist
We explore the problem of imitation learning (IL) in the context of
mean-field games (MFGs), where the goal is to imitate the behavior of a
population of agents following a Nash equilibrium policy according to some
unknown payoff function. IL in MFGs presents new challenges compared to
single-agent IL, particularly when both the reward function and the transition
kernel depend on the population distribution. In this paper, departing from the
existing literature on IL for MFGs, we introduce a new solution concept called
the Nash imitation gap. Then we show that when only the reward depends on the
population distribution, IL in MFGs can be reduced to single-agent IL with
similar guarantees. However, when the dynamics is population-dependent, we
provide a novel upper-bound that suggests IL is harder in this setting. To
address this issue, we propose a new adversarial formulation where the
reinforcement learning problem is replaced by a mean-field control (MFC)
problem, suggesting progress in IL within MFGs may have to build upon MFC.