An obstacle that prevents the wide adoption of (deep) reinforcement learning (RL) in control systems is its need for a large number of interactions with the environment in order to master a skill. The learned skill usually generalizes poorly across domains and re-training is often necessary when presented with a new task. We present a framework that combines techniques in \textit{formal methods} with \textit{hierarchical reinforcement learning} (HRL). The set of techniques we provide allows for the convenient specification of tasks with logical expressions, learns hierarchical policies (meta-controller and low-level controllers) with well-defined intrinsic rewards using any RL methods and is able to construct new skills from existing ones without additional learning. We evaluate the proposed methods in a simple grid world simulation as well as simulation on a Baxter robot. Reinforcement learning has received much attention in the recent years because of its achievements in games BID17 , , robotics manipulation Jang et al., , BID5 and autonomous driving BID8 , BID16 . However, training a policy that sufficiently masters a skill requires an enormous amount of interactions with the environment and acquiring such experience can be difficult on physical systems. Moreover, most learned policies are tailored to mastering one skill (by maximizing the reward) and are hardly reusable on a new skill.Skill composition is the idea of constructing new skills out of existing skills (and hence their policies) with little to no additional learning. In stochastic optimal control, this idea has been adopted by authors of BID25 and BID4 to construct provably optimal control laws based on linearly solvable Markov decision processes. Authors of BID6 , Tang & Haarnoja have showed in simulated manipulation tasks that approximately optimal policies can result from adding the Q-functions of the existing policies.Hierarchical reinforcement learning is an effective means of achieving transfer among tasks. The goal is to obtain task-invariant low-level policies, and by re-training the meta-policy that schedules over the low-level policies, different skills can be obtain with less samples than training from scratch. Authors of BID7 have adopted this idea in learning locomotor controllers and have shown successful transfer among simulated locomotion tasks. Authors of BID18 have utilized a deep hierarchical architecture for multi-task learning using natural language instructions.Temporal logic is a formal language commonly used in software and digital circuit verification BID1 as well as formal synthesis BID2 . It allows for convenient expression of complex behaviors and causal relationships. TL has been used by BID19 , BID13 to synthesize provably correct control policies. Authors of BID0 have also combined TL with Q-learning to learn satisfiable policies in discrete state and action spaces.In this work, we focus on hierarchical skill acquisition and zero-shot skill composition. Once a set of skills is acquired, we provide a technique that can synthesize new skills without the need to further interact with the environment (given the state and action spaces as well as the transition remain the same). We adopt temporal logic as the task specification language. Compared to most heuristic reward structures used in the RL literature to specify tasks, formal specification language excels at its semantic rigor and interpretability of specified behaviors. Our main contributions are:• We take advantage of the transformation between TL formula and finite state automata (FSA) to construct deterministic meta-controllers directly from the task specification without the necessity for additional learning. We show that by adding one discrete dimension to the original state space, structurally simple parameterized policies such as feed-forward neural networks can be used to learn tasks that require complex temporal reasoning.• Intrinsic motivation has been shown to help RL agents learn complicated behaviors with less interactions with the environment BID22 , BID11 BID9 . However , designing a well-behaved intrinsic reward that aligns with the extrinsic reward takes effort and experience. In our work, we construct intrinsic rewards directly from the input alphabets of the FSA (a component of the automaton), which guarantees that maximizing each intrinsic reward makes positive progress towards satisfying the entire task specification. From a user's perspective, the intrinsic rewards are constructed automatically from the TL formula.• In our framework, each FSA represents a hierarchical policy with low-level controllers that can be re-modulated to achieve different tasks. Skill composition is achieved by manipulating the FSA that results from their TL specifications in a deterministic fashion. Instead of interpolating/extrapolating among existing skills, we present a simple policy switching scheme based on graph manipulation of the FSA. Therefore, the compositional outcome is much more transparent. We introduce a method that allows learning of such hierarchical policies with any non-hierarchical RL algorithm. Compared with previous work on skill composition , we impose no constraints on the policy representation or the problem class. In this paper, we proposed the FSA augmented MDP, a product MDP that enables effective learning of hierarchical policies using any RL algorithm for tasks specified by scTLTL. We also introduced automata guided skill composition, a technique that combines existing skills to create new skills without additional learning. We show in robotic simulations that using the proposed methods we enable simple policies to perform logically complex tasks.Limitations of the current framework include discontinuity at the point of switching (for Equation (15)), which makes this method suitable for high level decision tasks but not for low level control tasks. The technique only compares robustness at the current step and chooses to follow a sub-policy for one time-step, making the switching policy short-sighted and may miss long term opportunities. One way to address this is to impose a termination condition for following each sub-policy and terminate only when the condition is triggered (as in the original options framework). This termination condition can be hand designed or learned
Combine temporal logic with hierarchical reinforcement learning for skill composition The paper offers a strategy for constructing a product MDP out of an original MDP and the automaton associated with an LTL formula. Proposes to join temporal logic with hierarchical reinforcement learning to simplify skill composition.