Maximum Causal Entropy Inverse Reinforcement Learning for Mean-Field
  Games
Berkay Anahtarci Can Deha Kariksiz Naci Saldi
In this paper, we introduce the maximum casual entropy Inverse Reinforcement
Learning (IRL) problem for discrete-time mean-field games (MFGs) under an
infinite-horizon discounted-reward optimality criterion. The state space of a
typical agent is finite. Our approach begins with a comprehensive review of the
maximum entropy IRL problem concerning deterministic and stochastic Markov
decision processes (MDPs) in both finite and infinite-horizon scenarios.
Subsequently, we formulate the maximum casual entropy IRL problem for MFGs - a
non-convex optimization problem with respect to policies. Leveraging the linear
programming formulation of MDPs, we restructure this IRL problem into a convex
optimization problem and establish a gradient descent algorithm to compute the
optimal solution with a rate of convergence. Finally, we present a new
algorithm by formulating the MFG problem as a generalized Nash equilibrium
problem (GNEP), which is capable of computing the mean-field equilibrium (MFE)
for the forward RL problem. This method is employed to produce data for a
numerical example. We note that this novel algorithm is also applicable to
general MFE computations.