ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Games
Shiqi Lei Kanghoon Lee Linjing Li Jinkyoo Park Jiachen Li
Offline learning has become widely used due to its ability to derive
effective policies from offline datasets gathered by expert demonstrators
without interacting with the environment directly. Recent research has explored
various ways to enhance offline learning efficiency by considering the
characteristics (e.g., expertise level or multiple demonstrators) of the
dataset. However, a different approach is necessary in the context of zero-sum
games, where outcomes vary significantly based on the strategy of the opponent.
In this study, we introduce a novel approach that uses unsupervised learning
techniques to estimate the exploited level of each trajectory from the offline
dataset of zero-sum games made by diverse demonstrators. Subsequently, we
incorporate the estimated exploited level into the offline learning to maximize
the influence of the dominant strategy. Our method enables interpretable
exploited level estimation in multiple zero-sum games and effectively
identifies dominant strategy data. Also, our exploited level augmented offline
learning significantly enhances the original offline learning algorithms
including imitation learning and offline reinforcement learning for zero-sum
games.