Goal-Conditioned Supervised Learning with Sub-Goal Prediction
Tom Jurgenson Aviv Tamar
Recently, a simple yet effective algorithm -- goal-conditioned
supervised-learning (GCSL) -- was proposed to tackle goal-conditioned
reinforcement-learning. GCSL is based on the principle of hindsight learning:
by observing states visited in previously executed trajectories and treating
them as attained goals, GCSL learns the corresponding actions via supervised
learning. However, GCSL only learns a goal-conditioned policy, discarding other
information in the process. Our insight is that the same hindsight principle
can be used to learn to predict goal-conditioned sub-goals from the same
trajectory. Based on this idea, we propose Trajectory Iterative Learner
(TraIL), an extension of GCSL that further exploits the information in a
trajectory, and uses it for learning to predict both actions and sub-goals. We
investigate the settings in which TraIL can make better use of the data, and
discover that for several popular problem settings, replacing real goals in
GCSL with predicted TraIL sub-goals allows the agent to reach a greater set of
goal states using the exact same data as GCSL, thereby improving its overall
performance.