ComSD: Balancing Behavioral Quality and Diversity in Unsupervised Skill
  Discovery
Xin Liu Yaran Chen Dongbin Zhao
Learning diverse and qualified behaviors for utilization and adaptation
without supervision is a key ability of intelligent creatures. Ideal
unsupervised skill discovery methods are able to produce diverse and qualified
skills in the absence of extrinsic reward, while the discovered skill set can
efficiently adapt to downstream tasks in various ways. Maximizing the Mutual
Information (MI) between skills and visited states can achieve ideal
skill-conditioned behavior distillation in theory. However, it's difficult for
recent advanced methods to well balance behavioral quality (exploration) and
diversity (exploitation) in practice, which may be attributed to the
unreasonable MI estimation by their rigid intrinsic reward design. In this
paper, we propose Contrastive multi-objectives Skill Discovery (ComSD) which
tries to mitigate the quality-versus-diversity conflict of discovered behaviors
through a more reasonable MI estimation and a dynamically weighted intrinsic
reward. ComSD proposes to employ contrastive learning for a more reasonable
estimation of skill-conditioned entropy in MI decomposition. In addition, a
novel weighting mechanism is proposed to dynamically balance different entropy
(in MI decomposition) estimations into a novel multi-objective intrinsic
reward, to improve both skill diversity and quality. For challenging robot
behavior discovery, ComSD can produce a qualified skill set consisting of
diverse behaviors at different activity levels, which recent advanced methods
cannot. On numerical evaluations, ComSD exhibits state-of-the-art adaptation
performance, significantly outperforming recent advanced skill discovery
methods across all skill combination tasks and most skill finetuning tasks.
Codes will be released at https://github.com/liuxin0824/ComSD.