Diversify & Conquer: Outcome-directed Curriculum RL via
  Out-of-Distribution Disagreement
Daesol Cho Seungjae Lee H. Jin Kim
Reinforcement learning (RL) often faces the challenges of uninformed search
problems where the agent should explore without access to the domain knowledge
such as characteristics of the environment or external rewards. To tackle these
challenges, this work proposes a new approach for curriculum RL called
Diversify for Disagreement & Conquer (D2C). Unlike previous curriculum learning
methods, D2C requires only a few examples of desired outcomes and works in any
environment, regardless of its geometry or the distribution of the desired
outcome examples. The proposed method performs diversification of the
goal-conditional classifiers to identify similarities between visited and
desired outcome states and ensures that the classifiers disagree on states from
out-of-distribution, which enables quantifying the unexplored region and
designing an arbitrary goal-conditioned intrinsic reward signal in a simple and
intuitive way. The proposed method then employs bipartite matching to define a
curriculum learning objective that produces a sequence of well-adjusted
intermediate goals, which enable the agent to automatically explore and conquer
the unexplored region. We present experimental results demonstrating that D2C
outperforms prior curriculum RL methods in both quantitative and qualitative
aspects, even with the arbitrarily distributed desired outcome examples.