Reconciling Spatial and Temporal Abstractions for Goal Representation
Mehdi Zadem Sergio Mover Sao Mai Nguyen
Goal representation affects the performance of Hierarchical Reinforcement
Learning (HRL) algorithms by decomposing the complex learning problem into
easier subtasks. Recent studies show that representations that preserve
temporally abstract environment dynamics are successful in solving difficult
problems and provide theoretical guarantees for optimality. These methods
however cannot scale to tasks where environment dynamics increase in complexity
i.e. the temporally abstract transition relations depend on larger number of
variables. On the other hand, other efforts have tried to use spatial
abstraction to mitigate the previous issues. Their limitations include
scalability to high dimensional environments and dependency on prior knowledge.
  In this paper, we propose a novel three-layer HRL algorithm that introduces,
at different levels of the hierarchy, both a spatial and a temporal goal
abstraction. We provide a theoretical study of the regret bounds of the learned
policies. We evaluate the approach on complex continuous control tasks,
demonstrating the effectiveness of spatial and temporal abstractions learned by
this approach.