We address the problem of learning structured policies for continuous control. In traditional reinforcement learning, policies of agents are learned by MLPs which take the concatenation of all observations from the environment as input for predicting actions. In this work, we propose NerveNet to explicitly model the structure of an agent, which naturally takes the form of a graph. Specifically, serving as the agent's policy network, NerveNet first propagates information over the structure of the agent and then predict actions for different parts of the agent. In the experiments, we first show that our NerveNet is comparable to state-of-the-art methods on standard MuJoCo environments. We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer. We demonstrate that policies learned by NerveNet are significantly better than policies learned by other models and are able to transfer even in a zero-shot setting.
 Deep reinforcement learning (RL) has received increasing attention over the past few years, with the recent success of applications such as playing Atari Games, Mnih et al. (2015) , and Go, BID26 . Significant advances have also been made in robotics using the latest RL techniques, e.g., BID1 ; BID19 .Many RL problems feature agents with multiple dependent controllers. For example, humanoid robots consist of multiple physically linked joints. Action to be taken by each joint or the body should thus not only depend on its own observations but also on actions of other joints.Previous approaches in RL typically use MLP to learn the agent's policy. In particular , MLP takes the concatenation of observations from the environment as input, which may be measurements like positions, velocities of body and joints in the current time instance. The MLP policy then predicts actions to be taken by every joint and body. Thus the task of the MLP policy is to discover the latent relationships between observations. This typically leads to longer training times, requiring more exposure of the agent to the environment. In our work, we aim to exploit the body structure of an agent, and physical dependencies that naturally exist in such agents.We rely on the fact that bodies of most robots and animals have a discrete graph structure. Nodes of the graph may represent the joints, and edges represent the (physical) dependencies between them. In particular, we define the agent's policy using a Graph Neural Network, Scarselli et al. (2009) , which is a neural network that operates over graph structures. We refer to our model as NerveNet due to the resemblance of the neural nervous system to a graph. NerveNet propagates information between different parts of the body based on the underlying graph structure before outputting the action for each part. By doing so, NerveNet can leverage the structure information encoded by the agent's body which is advantageous in learning the correct inductive bias, and thus is less prone to Figure 1: Visualization of the graph structure of CentipedeEight in our environment. We use this agent for testing the ability of transfer learning of our model. Since for this agent, each body node is paired with at least one joint node, we omit the body nodes and fill up the position with the corresponding joint nodes. By omitting the body nodes, a more compact graph is constructed, the details of which are illustrated in the experimental section.overfitting. Moreover, NerveNet is naturally suitable for structure transferring tasks as most of the model weights are shared across the nodes and edges, respectively.We first evaluate our NerveNet on standard RL benchmarks such as the OpenAI Gym, BID5 which stem from MuJoCo. We show that our model achieves comparable results to state-of-the-art MLP based methods. To verify our claim regarding the structure transfer , we further introduce our customized RL environments which are based on the ones of Gym. Two types of structure transfer tasks are designed, size transfer and disability transfer. In particular, size transfer focuses on the scenario in which policies are learned for small-sized agents (simpler body structure) and applied directly to large-sized agents which are composed by some repetitive components shared with the small-sized agent. Secondly, disability transfer investigates scenarios in which policies are learned for one agent and applied to the same agent with some components disabled. Our experiments demonstrate that for structure transfer tasks our NerveNet is significantly better than all other competitors, and can even achieve zero-shot learning for some agents. For the multi-task learning tasks, NerveNet is also able to learn policies that are more robust with better efficiency.The main contribution of this paper is the following: We explore the problem of learning transferable and generalized features by incorporating a prior on the structure via graph neural networks. NerveNet permits powerful transfer learning from one structure to another, which goes well beyond the ability of previous models. NerveNet is also more robust and has more potential in performing multi-task learning. The demo and code for this project are released, under the project page of http://www.cs.toronto.edu/˜tingwuwang/nervenet.html. In this paper, we aimed to exploit the body structure of Reinforcement Learning agents in the form of graphs. We introduced a novel model called NerveNet which uses a Graph Neural Network to represent the agent's policy. At each time instance of the environment, NerveNet takes observations for each of the body joints, and propagates information between them using non-linear messages computed with a neural network. Propagation is done through the edges which represent natural dependencies between joints, such as physical connectivity. We experimentally showed that our NerveNet achieves comparable performance to state-of-the-art methods on standard MuJoCo environments. We further propose our customized reinforcement learning environments for benchmarking two types of structure transfer learning tasks, i.e., size and disability transfer. We demonstrate that policies learned by NerveNet are significantly better than policies learned by other models and are able to transfer even in a zero-shot setting.
using graph neural network to model structural information of the agents to improve policy and transferability  A method for representing and learning structured policy for continuous control tasks using Graph Neural Networks The submission proposes incorporation of additional structure into reinforcement learning problems, particularly the structure of the agent's morphology Propose an application of Graph Neural Networks to learning policies for controlling "centipede" robots of different lengths.