Humans are experts at high-fidelity imitation -- closely mimicking a demonstration, often in one attempt. Humans use this ability to quickly solve a  task instance, and to bootstrap learning of new tasks. Achieving these abilities in autonomous agents is an open problem. In this paper, we introduce an off-policy RL algorithm (MetaMimic) to narrow this gap. MetaMimic can learn both (i) policies for high-fidelity one-shot imitation of diverse novel skills, and (ii) policies that enable the agent to solve tasks more efficiently than the demonstrators. MetaMimic relies on the principle of storing all experiences in a memory and replaying these to learn massive deep neural network policies by off-policy RL. This paper introduces, to the best of our knowledge, the largest existing neural networks for deep RL and shows that larger networks with normalization are needed to achieve one-shot high-fidelity imitation on a challenging manipulation task.
 The results also show that both types of policy can be learned from vision, in spite of the task rewards being sparse, and without access to demonstrator actions. One-shot imitation is a powerful way to show agents how to solve a task. For instance, one or a few demonstrations are typically enough to teach people how to solve a new manufacturing task. In this paper, we introduce an AI agent that when provided with a novel demonstration is able to (i) mimic the demonstration with high-fidelity, or (ii) forego high-fidelity imitation to solve the intended task more efficiently. Both types of imitation can be useful in different domains.Motor control is a notoriously difficult problem, and we are often deceived by how simple a manipulation task might appear to be. Tying shoe-laces, a behaviour many of us learn by imitation, might appear to be simple. Yet, tying shoe-laces is something most 6 year olds struggle with, long after object recognition, walking, speech, often translation, and sometimes even reading comprehension. This long process of learning that eventually results in our ability to rapidly imitate many behaviours provides inspiration for the work in this paper.We refer to high-fidelity imitation as the act of closely mimicking a demonstration trajectory, even when some actions may be accidental or irrelevant to the task. This is sometimes called over-imitation BID28 . It is known that humans over-imitate more than other primates BID18 and that this may be useful for rapidly acquiring new skills BID24 . For AI agents however, learning to closely imitate even one single demonstration from raw sensory input can be difficult. Many recent works focus on using expensive reinforcement learning (RL) methods to solve this problem BID46 BID27 BID37 BID3 . In contrast, high-fidelity imitation in humans is often cheap: in one-shot we can closely mimic a demonstration. Inspired by this, we introduce a meta-learning approach (MetaMimic - FIG0 ) to learn high-fidelity one-shot imitation policies by off-policy RL. These policies, when deployed, require a single demonstration as input in order to mimic the new skill being demonstrated.AI agents could acquire a large and diverse set of skills by high-fidelity imitation with RL. However, representing many behaviours requires the adoption of a model with very high capacity, such as a very large deep neural network. Unfortunately, showing that RL methods can be used to train massive deep neural networks has been an open question because of the variance inherent to these methods. Indeed, traditional deep RL neural networks tend to be small, to the point that researchers have recently questioned their contribution BID42 . In this paper, we show that it is possible to train massive high-fidelity imitation policy π(ot,gt) with off-policy RL. This policy, represented with a massive deep neural network, enables the robot arm to mimic any demonstration in one-shot. In addition to producing an imitation policy that generalizes well, MetaMimic populates its replay memory with all its rich experiences, including not only the demonstration videos, but also its past observations, actions and rewards. By harnessing these augmented experiences, a task policy π(ot) can be trained to solve difficult sparse-reward control tasks.deep networks by off-policy RL to represent many behaviours. Moreover, we show that bigger networks generalize better. These results therefore provide important evidence that RL is indeed a scalable and viable framework for the design of AI agents. Specifically this paper makes the following contributions 1 :• It introduces the MetaMimic algorithm and shows that it is capable of one-shot high-fidelity imitation from video in a complex manipulation domain.• It shows that MetaMimic can harness video demonstrations and enrich them with actions and rewards so as to learn uncoditional policies capable of solving manipulation tasks more efficiently than teleoperating humans. By retaining and taking advantage of all its experiences, MetaMimic also substantially outperforms the state-of-the-art D4PG RL agent, when D4PG uses only the current task experiences.• The experiments provide ablations showing that larger networks (to the best of our knowledge, the largest networks ever used in deep RL) lead to improved generalization in high-fidelity imitation. The ablations also highlight the important value of instance normalization.• The experiments show that increasing the number of demonstrations during training leads to better generalization on one-shot high-fidelity imitation tasks. In this paper, we introduced MetaMimic, a method to 1) train a high-fidelity one-shot imitation policy, and to 2) efficiently train a task policy. MetaMimic employs the largest neural network trained via RL, and works from vision, without the need of expert actions. The one-shot imitation policy can generalize to unseen trajectories and can mimic them closely. Bootstrapping on imitation experiences, the task policy can quickly outperform the demonstrator, and is competitive with methods that receive privileged information.The framework presented in this paper can be extended in a number of ways. First, it would be exciting to combine this work with existing methods for learning third-person imitation rewards BID45 BID3 . This would bring us a step closer to how humans imitate: By watching other agents act in the environment. Second, it would be exciting to extend MetaMimic to imitate demonstrations of a variety of tasks. This may allow it to generalize to demonstrations of unseen tasks.To improve the ease of application of MetaMimic to robotic tasks, it would be desirable to address the question of how to relax the initialization constraints for high-fidelity imitation; specifically not having to set the initial agent observation to be close to the initial demonstration observation.
We present MetaMimic, an algorithm that takes as input a demonstration dataset and outputs (i) a one-shot high-fidelity imitation policy (ii) an unconditional task policy. The paper looks at the problem of one-shot imitation with high accuracy of imitation, extending DDPGfD to use only state trajectories. This paper proposes an approach for one-shot imitation with high accuracy, and addresses the common problem of exploration in imitation learning. Presents an RL method for learning from video demonstration without access to expert actions