Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning
Anna Penzkofer Simon Schaefer Florian Strohm Mihai Bâce Stefan Leutenegger Andreas Bulling
While deep reinforcement learning (RL) agents outperform humans on an
increasing number of tasks, training them requires data equivalent to decades
of human gameplay. Recent hierarchical RL methods have increased sample
efficiency by incorporating information inherent to the structure of the
decision problem but at the cost of having to discover or use human-annotated
sub-goals that guide the learning process. We show that intentions of human
players, i.e. the precursor of goal-oriented decisions, can be robustly
predicted from eye gaze even for the long-horizon sparse rewards task of
Montezuma's Revenge - one of the most challenging RL tasks in the Atari2600
game suite. We propose Int-HRL: Hierarchical RL with intention-based sub-goals
that are inferred from human eye gaze. Our novel sub-goal extraction pipeline
is fully automatic and replaces the need for manual sub-goal annotation by
human experts. Our evaluations show that replacing hand-crafted sub-goals with
automatically extracted intentions leads to a HRL agent that is significantly
more sample efficient than previous methods.