In the pursuit of increasingly intelligent learning systems, abstraction plays a vital role in enabling sophisticated decisions to be made in complex environments. The options framework provides formalism for such abstraction over sequences of decisions.   However most models require that options be given a priori, presumably specified by hand, which is neither efficient, nor scalable. Indeed, it is preferable to learn options directly from interaction with the environment. Despite several efforts, this remains a difficult problem: many approaches require access to a model of the environmental dynamics, and inferred options are often not interpretable, which limits our ability to explain the system behavior for verification or debugging purposes.   In this work we develop a novel policy gradient method for the automatic learning of policies with options.   This algorithm uses inference methods to simultaneously improve all of the options available to an agent, and thus can be employed in an off-policy manner, without observing option labels. Experimental results show that the options learned can be interpreted. Further, we find that the method presented here is more sample efficient than existing methods, leading to faster and more stable learning of policies with options. Recent developments in reinforcement learning (RL) methods have enabled agents to solve problems in increasingly complicated domains BID14 . However, in order for agents to solve more difficult and realistic environments-potentially involving long sequences of decisions-more sample efficient techniques are needed. One way to improve on existing agents is to leverage abstraction. By reasoning at various levels of abstraction, it is possible to infer, learn and plan much more efficiently. Recent developments have lead to breakthroughs in terms of learning rich representations for perceptual information BID2 . In RL domains, however, while efficient methods exist to plan and learn when abstraction over sequences of actions is provided a priori, it has proven more difficult to learn this type of temporal abstraction from interaction data.Many frameworks for formalizing temporal abstraction have been proposed; most recent developments build on the Options framework BID21 BID16 , which offers a flexible parameterization, potentially amenable to learning. The majority of prior work on learning options has centered around the idea of discovering subgoals in state space, and constructing a set of options such that each option represents a policy leading to that subgoal BID11 BID12 BID20 ). These methods can lead to useful abstraction, however, they often require access to a model of the environment dynamics, which is not always available, and can be infeasible to learn. Our contributions instead build on the work of BID1 , and exploits a careful parameterization of the policy of the agent, in order to simultaneously learn a set of options, while directly optimizing returns. We relax a few key assumptions of this previous work, including the expectation that only options that were actually executed during training can be learned, and the focus on executing options in an on-policy manner, with option labels available. By relaxing these, we can improve sample efficiency and practical applicability, including the possibility to seed control policies from expert demonstrations. In this paper, we have introduced a new algorithm for learning hierarchical policies within the options framework, called inferred option policy gradients. This algorithm treats options as latent variables. Gradients are propagated through a differentiable inference step that allows end-to-end learning of option policies, as well as option selection and termination probabilities.In our algorithms policies take responsibility for state-actions pairs they could have generated. In contrast, in learning algorithms for hierarchical policies that use an augmented state space, option policies are updated using only those state-action pairs the actually generated. As a result, in our algorithm options do not tend to become 'responsible' for unlikely states or actions they generated. Thus, options are stimulated more strongly to specialize in a part of the state space. We conjecture that this specialization caused the discussed increase in the interpretability of options.(a ) Performance in the Walker2d environment as a function of available options. We see that in this environment, having several options available to the agent leads to an improved policy.(b) In the Walker2d environment, initially option selection is uniform. After training only 3 options tend to be selected, even when more are available. Selection frequencies are averaged over 100 sampled states.(c) As the options improve during training, the probability of remaining in the active option increases, plateauing at around 0.85. This suggests that the options learned here exhibit temporal extension. Furthermore, in our experiments learning with inferred options was significantly faster than learning with an option-augmented state space. In fact, learning with inferred options proved equally fast, or sometimes even faster, than using a comparable non-hierarchical policy gradient method despite IOPG having many more parameters. We conjecture that option inference encourages intra-option learning, thus allowing multiple options to improve as the result of a single learning experience, causing this speed-up.In future work, we want to quantify the suitability of the learned options for transfer between tasks. Our experiments so far were in the episodic setting. We want to investigate an on-line, actor-critic version of learning with inferred options to learn continuously in infinite-horizon problems.
We develop a novel policy gradient method for the automatic learning of policies with options using a differentiable inference step. The paper presents a new policy gradient technique for learning options, where a single sample can be used to update all options. Proposes an off-policy method for learning options in complex continuous problems.