Skills learned through (deep) reinforcement learning often generalizes poorly
 across tasks and re-training is necessary when presented with a new task. We
 present a framework that combines techniques in formal methods with reinforcement
 learning (RL) that allows for the convenient specification of complex temporal
 dependent tasks with logical expressions and construction of new skills from existing
 ones with no additional exploration. We provide theoretical results for our
 composition technique and evaluate on a simple grid world simulation as well as
 a robotic manipulation task. Policies learned using reinforcement learning aim to maximize the given reward function and are often difficult to transfer to other problem domains. Skill composition is the process of constructing new skills out of existing ones (policies) with little to no additional learning. In stochastic optimal control, this idea has been adopted by BID20 and BID9 to construct provably optimal control laws based on linearly solvable Markov decision processes.Temporal logic (TL) is a formal language commonly used in software and digital circuit verification BID7 as well as formal synthesis BID8 . It allows for convenient expression of complex behaviors and causal relationships. TL has been used by BID19 , BID11 , BID10 to synthesize provably correct control policies. BID6 have also combined TL with Q-learning to learn satisfiable policies in discrete state and action spaces.We make the distinction between skill composition and multi-task learning/meta-learning where the latter often requires a predefined set of tasks/task distributions to learn and generalize from, whereas the focus of the former is to construct new policies from a library of already learned policies that achieve new tasks (often some combination of the constituent tasks) with little to no additional constraints on task distribution at learning time. In this work, we focus on skill composition with policies learned using automata guided reinforcement learning BID15 . We adopt the syntactically co-safe truncated linear temporal logic (scTLTL) as the task specification language. Compared to most heuristic reward structures used in the RL literature, formal specification language has the advantage of semantic rigor and interpretability.In our framework, skill composition is accomplished by taking the product of finite state automata (FSA). Instead of interpolating/extrapolating among learned skills/latent features, our method is based on graph manipulation of the FSA. Therefore, the outcome is much more transparent. Compared with previous work on skill composition, we impose no constraints on the policy representation or the problem class. We validate our framework in simulation (discrete state and action spaces) and experimentally on a Baxter robot (continuous state and action spaces). In FIG6 (left), we report the discounted return as a function of policy update steps for task φ ∧ . 5 evaluation episodes are collected after each set policy updates to calculate the performance statistics. As comparison, we learn the same task using SQL with FSA augmented MDP. We can see that our composition method takes less update steps to reach a policy that achieves higher returns with lower variance than the policy obtained from learning. FIG6 (right) shows the episode length as a function of policy update (upper bound clipped at 100 steps). As mentioned in the previous section, a shorter episode length indicates faster accomplishment of the task. It can be observed that both the composition and learning method result in high variances likely due to the randomized task configuration (some plate/joint/hand configurations make the task easier to accomplish than others). However, the policy obtained from composition achieves a noticeable decrease in the average episode length.It is important to note that the wall time for learning a policy is significantly longer than that from composition. For robotic tasks with relatively simple policy representations (feed-forward neural networks), learning time is dominated by the time used to collect experiences and the average episode length (recall that we update the policy 100 times with each 5 episodes of exploration). Since skill composition uses already collected experience, obtaining a policy can be much faster. TAB1 shows the mean training time and standard deviation (over 5 random seeds) for each task (tasks φ traverse , φ interrupt and φ ∧ (learned) are trained for 80K policy updates. φ ∧ (composed) is trained for 40K policy updates). In general, training time is shorter for tasks with higher episodic success rate and shorter episode length. We also show the task success rate evaluated on the real robot over 20 evaluation trials. Task success is evaluated by calculating the robustness of the trajectories resulting from executing each policy. A robustness of greater than 0 evaluates to success and vice versa. π φ∧ (learned) fails to complete the task even though a convergence is reached during training. This is likely due to the large FSA of φ ∧ with complex per-step reward (D q φ in Equation FORMULA15 ) which makes learning difficult. FIG5 shows an evaluation run of the composed policy for task φ ∧ . We provide a technique that takes advantage of the product of finite state automata to perform deterministic skill composition. Our method is able to synthesize optimal composite policies for −AN D− and −OR− tasks. We provide theoretical results on our method and show its effectiveness on a grid world simulation and a real world robotic task. For future work, we will adapt our method to the more general case of task-space transfer -given a library of optimal policies (Qfunctions) that each satisfies its own specification, construct a policy that satisfies a specification that's an arbitrary (temporal) logical combination of the constituent specifications.
A formal method's approach to skill composition in reinforcement learning tasks The paper combines RL and constraints expressed by logical formulas by setting up an automation from scTLTL formulas. Proposes a method that helps to construct policy from learned subtasks on the topic of combining RL tasks with linear temporal logic formulas.