Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a novel framework for efficient multi-task reinforcement learning. Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill. This enables agents to continually acquire new skills during different stages of training. Each learned task corresponds to a human language description. Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices. In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills. We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills. Deep reinforcement learning has demonstrated success in policy search for tasks in domains like game playing BID12 BID7 BID11 and robotic control BID9 b; BID16 . However, it is very difficult to accumulate multiple skills using just one policy network BID24 . Knowledge transfer techniques like distillation BID3 BID18 BID15 BID24 have been applied to train a policy network both to learn new skills while preserving previously learned skill as well as to combine single-task policies into a multi-task policy. Existing approaches usually treat all tasks independently. This often prevents full exploration of the underlying relations between different tasks. They also typically assume that all policies share the same state space and action space. This precludes transfer of previously learned simple skills to a new policy defined over a space with differing states or actions.When humans learn new skills, we often take advantage of our existing skills and build new capacities by composing or combining simpler ones. For instance, learning multi-digit multiplication relies on the knowledge of single-digit multiplication; learning how to properly prepare individual ingredients facilitates cooking dishes based on complex recipes.Inspired by this observation, we propose a hierarchical policy network which can reuse previously learned skills alongside and as subcomponents of new skills. It achieves this by discovering the underlying relations between skills.To represent the skills and their relations in an interpretable way, we also encode all tasks using human instructions such as "put down." This allows the agent to communicate its policy and generate plans using human language. Figure 1 illustrates an example: given the instruction "Stack blue," our hierarchical policy learns to compose instructions and take multiple actions through a multi-level hierarchy in order to stack two blue blocks together. Steps from the top-level policy π 3 (i.e., the red Figure 1: Example of our multi-level hierarchical policy for a given task -stacking two blue blocks. Each arrow represents one step generated by a certain policy and the colors of arrows indicate the source policies. Note that at each step, a policy either utters an instruction for the lower-level policy or directly takes an action.branches) outline a learned high-level plan -"Get blue → Find blue → Put blue." In addition, from lower level policies, we may also clearly see composed plans for other tasks. Based on policy π 2 , for instance, the task "Get blue" has two steps -"Find blue → action: turn left," whereas "Put blue" can be executed by a single action "put down" according to π 3 . Through this hierarchical model, we may i) accumulate tasks progressively from a terminal policy to a top-level policy and ii) unfold the global policy from top-level to basic actions.In order to better track temporal relationships between tasks, we train a stochastic temporal grammar (STG) model on the sequence of policy selections (previously learned skill or new skill) for positive episodes. The STG focuses on modeling priorities of tasks: for example, it is necessary to obtain an object before putting it down. Integrating the STG into the hierarchical policy boosts efficiency and accuracy by explicitly modeling such commonsense world knowledge.We validated our approach by testing it on object manipulation tasks implemented in a Minecraft world. Our experimental results demonstrate that this framework can (i) efficiently learn hierarchical policies and representations for multi-task RL; (ii) learn to utter human instructions to deploy pretrained policies, improve their explainability and reuse skills; and (iii) learn a stochastic temporal grammar via self-supervision to predict future actions. In this work, we have proposed a hierarchal policy modulated by a stochastic temporal grammar as a novel framework for efficient multi-task reinforcement learning through multiple training stages. Each task in our settings is described by a human instruction. The resulting global policy is able to reuse previously learned skills for new tasks by generating corresponding human instructions to inform base policies to execute relevant base tasks. We evaluate this framework in Minecraft games and have shown that our full model i) has a significantly higher learning efficiency than a flat policy does, ii) generalizes well in unseen environments, and iii) is capable of composing hierarchical plans in an interpretable manner.Currently, we rely on weak supervision from humans to define what skills to be learned in each training stage. In the future, we plan to automatically discover the optimal training procedures to increase the task set.A PSEUDO CODE OF OUR ALGORITHMS Algorithm 1 RUN(k, g) Input: Policy level k, task g ∈ G k Output: Episode trajectory Γ at the top level policy 1: t ← 0 2: Γ = ∅ 3: Get initial state s0 4: repeat 5:if k == 1 then 6:Sample at ∼ π k (·|st, g) and execute at 7:Get current state st+1 8:rt ← R(st+1, g) 9:Add st, at, rt, π k (·|st, g), g to Γ 10: else 11:Sample et and g t as in Section 3.3 for using STG as guidance 12:Sample at ∼ π aug k (·|st, g) 13:if et = 0 then 14:// Execute base policy π k−1 by giving instruction g t 15:RUN(k − 1, g t ) 16: else 17:Execute at 18: end if 19:Get current state st+1 20:rt ← R(st+1, g) 21:Add st, et, g t , at, rt, π DISPLAYFORM0 end if 23: if in curriculum learning phase 1 then 9: DISPLAYFORM1 Sample a task g from base task set G k−1 10: else 11:Sample a task g from global task set G k 12:end if 13://Run an episode 14: DISPLAYFORM2 if the maximum reward in Γ is +1 then 17: DISPLAYFORM3 Re-estimate the distributions of the STG based on updated D+ by MLE 19:end if 20:Sample n ∼ Possion(λ) 21:for j ∈ {1, · · · , n} do 22:Sample a mini-batch S from D 23:Update Θ based on (9) and the τ -th term in (8) 24:i ← i + 1 25:if i%M = 0 then 26:τ ← τ %3 + 1 27: end if 28: end for 29: until i ≥ N
A novel hierarchical policy network which can reuse previously learned skills alongside and as subcomponents of new skills by discovering the underlying relations between skills. This paper aims to learn hierarchical policies by using a recursive policy structure regulated by a stochastic temporal grammar This paper proposes an approach to learning hierarchical policies in a lifelong learning context by stacking policies and then using an explicit "switch" policy.