ELDEN: Exploration via Local Dependencies
Jiaheng Hu Zizhao Wang Peter Stone Roberto Martin-Martin
Tasks with large state space and sparse rewards present a longstanding
challenge to reinforcement learning. In these tasks, an agent needs to explore
the state space efficiently until it finds a reward. To deal with this problem,
the community has proposed to augment the reward function with intrinsic
reward, a bonus signal that encourages the agent to visit interesting states.
In this work, we propose a new way of defining interesting states for
environments with factored state spaces and complex chained dependencies, where
an agent's actions may change the value of one entity that, in order, may
affect the value of another entity. Our insight is that, in these environments,
interesting states for exploration are states where the agent is uncertain
whether (as opposed to how) entities such as the agent or objects have some
influence on each other. We present ELDEN, Exploration via Local DepENdencies,
a novel intrinsic reward that encourages the discovery of new interactions
between entities. ELDEN utilizes a novel scheme -- the partial derivative of
the learned dynamics to model the local dependencies between entities
accurately and computationally efficiently. The uncertainty of the predicted
dependencies is then used as an intrinsic reward to encourage exploration
toward new interactions. We evaluate the performance of ELDEN on four different
domains with complex dependencies, ranging from 2D grid worlds to 3D robotic
tasks. In all domains, ELDEN correctly identifies local dependencies and learns
successful policies, significantly outperforming previous state-of-the-art
exploration methods.