Iteratively Learn Diverse Strategies with State Distance Information
Wei Fu Weihua Du Jingwei Li Sunli Chen Jingzhao Zhang Yi Wu
In complex reinforcement learning (RL) problems, policies with similar
rewards may have substantially different behaviors. It remains a fundamental
challenge to optimize rewards while also discovering as many diverse strategies
as possible, which can be crucial in many practical applications. Our study
examines two design choices for tackling this challenge, i.e., diversity
measure and computation framework. First, we find that with existing diversity
measures, visually indistinguishable policies can still yield high diversity
scores. To accurately capture the behavioral difference, we propose to
incorporate the state-space distance information into the diversity measure. In
addition, we examine two common computation frameworks for this problem, i.e.,
population-based training (PBT) and iterative learning (ITR). We show that
although PBT is the precise problem formulation, ITR can achieve comparable
diversity scores with higher computation efficiency, leading to improved
solution quality in practice. Based on our analysis, we further combine ITR
with two tractable realizations of the state-distance-based diversity measures
and develop a novel diversity-driven RL algorithm, State-based Intrinsic-reward
Policy Optimization (SIPO), with provable convergence properties. We
empirically examine SIPO across three domains from robot locomotion to
multi-agent games. In all of our testing environments, SIPO consistently
produces strategically diverse and human-interpretable policies that cannot be
discovered by existing baselines.