Recent state-of-the-art reinforcement learning algorithms are trained under the goal of excelling in one specific task. Hence, both environment and task specific knowledge are entangled into one framework. However, there are often scenarios where the environment (e.g. the physical world) is fixed while only the target task changes. Hence, borrowing the idea from hierarchical reinforcement learning, we propose a framework that disentangles task and environment specific knowledge by separating them into two units. The environment-specific unit handles how to move from one state to the target state; and the task-specific unit plans for the next target state given a specific task. The extensive results in simulators indicate that our method can efficiently separate and learn two independent units, and also adapt to a new task more efficiently than the state-of-the-art methods. Let's imagine ourselves learning how to play tennis for the first time. Even though we have never played tennis before, we already have a good understanding of agent and environment dynamics related to tennis. For example, we know how to move our arm from one position to another and that a ball will slow down and bounce back from the ground. Hence, we just need to learn the tennis specific knowledge (e.g. its game rule and a relationship between an arm control and a tennis racket). Just like this example, when we learn to complete a new task, we utilize the prior knowledge that is disentangled from the task and acquired over our lifetime. Figure 1: Our model disentangles environment-specific information (e.g. transition dynamics) and task-specific knowledge (e.g. task rewards) for training efficiency and interpretability.From a reinforcement learning perspective, this brings a very interesting question -how can agents also obtain and utilize such disentangled prior knowledge about the environment? Most of today's deep reinforcement learning (DRL) models BID14 ; BID25 are trained with entangled environment-specific knowledge (e.g. transition dynamics) and taskspecific knowledge (e.g. rewards), as described in Figure 1a However, as described earlier, humans τ τ Figure 2 : Proposed universal agent, which consists of three parts: a φ function mapping raw observation to feature space, a PATH function as an environment actor, and a τ function for future state planning.have an innate ability to obtain a good understanding about the environment dynamics, and utilize them in a newly given task. Motivated from this, we introduce a new scheme to disentangle the learning procedure of task-independent transition dynamics and task-specific rewards, as described in Figure 2 . This will help an agent to adapt to a new task more efficiently and also provides an extra interpretability.The idea of disentangling a model into two components can be related to hierarchical RL approaches BID29 ; BID17 ; BID16 . However, to the best of our knowledge, there has not been a work that separating units by the natural criteria of environment and task specific knowledge, for the goal of transfer learning.To this end, we introduce a model that consists of two major units: a PATH function and a goal generator. This is illustrated in Figure 2 . The key intuition is as the following. PATH function handles the environment specific knowledge, and a goal function handles the task specific knowledge. We design (1) PATH function to learn how to move from one state to another -a lower-level controller, which is independent from the task and only depends on the environment, and (2) the goal function τ to determine the next state given a target task -a higher-level planner. Thus, PATH function can be shared across different tasks as long as it is under the same environment (e.g. the physical world).We evaluate our method to answer the following two questions: (1) how a good PATH unit can benefit the task learning, and (2) how efficient is our model for learning a new task in the same environment. We analyze the behavior of our method on various environments including a maze world and multiple Atari 2600 games. Our study shows that a good PATH unit can be trained, and our model has a faster convergence compared to the state-of-the-art method BID15 in most of the tasks, especially on transfer learning tasks.In summary, we introduce an RL model with disentangled units for task-specific and environmentspecific knowledge. We demonstrate in multiple environments that our method learns environmentspecific knowledge, which further enables an agent adapting to a new task in the same environment. Model-based reinforcement learning Typical model-based RL frameworks aim to model the dynamics of the environment. They usually involve search-based algorithms (e.g., Monte Carlo Tree Search) as part of the policy Sutton & Barto (1998) (e.g., Alpha GO with a simulator , Scheme-networks with learned forward dynamics BID11 ). In this paper, we address the problem of representing the knowledge about the environment by learning skills (how to reach a given state). As a cognitive science motivation described in BID6 , humans also store the knowledge of movements or actions by their end-states. Most importantly, this knowledge can be easily utilized by a task-specific module (the goal generator) to exploit novel tasks.Multi-task learning and transfer learning Multi-task learning and transfer learning BID2 ; BID0 ; Taylor & Stone (2009) provide approaches to transfer knowledge among multiple agents. The methods include the decomposition of value function or task and direct multi-task learning where agents learn several tasks jointly. Contrary to them, universal agent is able to obtain the environment specific knowledge without any specific task supervision. (2015) ; BID32 , the τ function does not perform option selection but directly composes target state since a general-purpose PATH function is utilized. We do not follow typical HRL methods where once the subtask is selected, the low-level controller will be executed for multiple time steps. In universal agent, for simplicity now, τ function plans the future state at every time step. We leave its adaption to HRL frameworks as a future work. We present a new reinforcement learning scheme that disentangles the learning procedure of taskindependent transition dynamics and task-specific rewards. The main advantage of this is efficiency of task adaptation and interpretability. For this we simply introduce two major units: a PATH function and a goal generator. Our study shows that a good PATH unit can be trained, and our model outperforms the state-of-the-art method BID15 in most of tasks, especially on transfer learning tasks.The proposed framework is a novel step towards the knowledge representation learning for deep reinforcement learning (DRL). There are a variety of future research directions. For example, how PATH function can be learned (e.g., in a continual learning manner BID22 , with less requirement such as state restoration), how it can better cooperate with the goal generator (e.g., incorporating explicit future planning) and how it can be used for other tasks (e.g., learning from demonstration).
We propose a DRL framework that disentangles task and environment specific knowledge. The authors propose to decompose reinforcement learning into a PATH function and a GOAL function A modular architecture with the aim of separating environment specific knowledge and task-specific knowledge into different modules, on par with standard A3C across a wide range of tasks.