Behavior Contrastive Learning for Unsupervised Skill Discovery
Rushuai Yang Chenjia Bai Hongyi Guo Siyuan Li Bin Zhao Zhen Wang Peng Liu Xuelong Li
In reinforcement learning, unsupervised skill discovery aims to learn diverse
skills without extrinsic rewards. Previous methods discover skills by
maximizing the mutual information (MI) between states and skills. However, such
an MI objective tends to learn simple and static skills and may hinder
exploration. In this paper, we propose a novel unsupervised skill discovery
method through contrastive learning among behaviors, which makes the agent
produce similar behaviors for the same skill and diverse behaviors for
different skills. Under mild assumptions, our objective maximizes the MI
between different behaviors based on the same skill, which serves as an upper
bound of the previous MI objective. Meanwhile, our method implicitly increases
the state entropy to obtain better state coverage. We evaluate our method on
challenging mazes and continuous control tasks. The results show that our
method generates diverse and far-reaching skills, and also obtains competitive
performance in downstream tasks compared to the state-of-the-art methods.