Wasserstein Diversity-Enriched Regularizer for Hierarchical
  Reinforcement Learning
Haorui Li Jiaqi Liang Linjing Li Daniel Zeng
Hierarchical reinforcement learning composites subpolicies in different
hierarchies to accomplish complex tasks.Automated subpolicies discovery, which
does not depend on domain knowledge, is a promising approach to generating
subpolicies.However, the degradation problem is a challenge that existing
methods can hardly deal with due to the lack of consideration of diversity or
the employment of weak regularizers. In this paper, we propose a novel
task-agnostic regularizer called the Wasserstein Diversity-Enriched Regularizer
(WDER), which enlarges the diversity of subpolicies by maximizing the
Wasserstein distances among action distributions. The proposed WDER can be
easily incorporated into the loss function of existing methods to boost their
performance further.Experimental results demonstrate that our WDER improves
performance and sample efficiency in comparison with prior work without
modifying hyperparameters, which indicates the applicability and robustness of
the WDER.