Co-Learning Empirical Games and World Models
Max Olan Smith Michael P. Wellman
Game-based decision-making involves reasoning over both world dynamics and
strategic interactions among the agents. Typically, empirical models capturing
these respective aspects are learned and used separately. We investigate the
potential gain from co-learning these elements: a world model for dynamics and
an empirical game for strategic interactions. Empirical games drive world
models toward a broader consideration of possible game dynamics induced by a
diversity of strategy profiles. Conversely, world models guide empirical games
to efficiently discover new strategies through planning. We demonstrate these
benefits first independently, then in combination as realized by a new
algorithm, Dyna-PSRO, that co-learns an empirical game and a world model. When
compared to PSRO -- a baseline empirical-game building algorithm, Dyna-PSRO is
found to compute lower regret solutions on partially observable general-sum
games. In our experiments, Dyna-PSRO also requires substantially fewer
experiences than PSRO, a key algorithmic advantage for settings where
collecting player-game interaction data is a cost-limiting factor.