The goal of imitation learning (IL) is to enable a learner to imitate an expert’s behavior given the expert’s demonstrations. Recently, generative adversarial imitation learning (GAIL) has successfully achieved it even on complex continuous control tasks. However, GAIL requires a huge number of interactions with environment during training. We believe that IL algorithm could be more applicable to the real-world environments if the number of interactions could be reduced. To this end, we propose a model free, off-policy IL algorithm for continuous control. The keys of our algorithm are two folds: 1) adopting deterministic policy that allows us to derive a novel type of policy gradient which we call deterministic policy imitation gradient (DPIG), 2) introducing a function which we call state screening function (SSF) to avoid noisy policy updates with states that are not typical of those appeared on the expert’s demonstrations. Experimental results show that our algorithm can achieve the goal of IL with at least tens of times less interactions than GAIL on a variety of continuous control tasks. Recent advances in reinforcement learning (RL) have achieved super-human performance on several domains BID16 BID17 BID13 . However, on most of domains with the success of RL, the design of reward, that explains what agent's behavior is favorable, is clear enough for humans. Conversely, on domains where it is unclear how to design the reward, agents trained by RL algorithms often obtain poor policies and their behavior is far from what we want them to do. Imitation learning (IL) comes in such cases. The goal of IL is to enable the learner to imitate the expert's behavior given the expert's demonstrations but the reward signals. We are interested in IL because we desire an algorithm that can be applied in real-world environments where it is often hard to design the reward. Besides, since it is generally hard to model a variety of the real-world environments with an algorithm, and the state-action pairs in a vast majority of the real-world applications such as robotics control can be naturally represented in continuous spaces, we focus on model free IL for continuous control.A widely used approach of existing model free IL methods is the combination of Inverse Reinforcement Learning (IRL) BID22 BID18 BID1 BID32 and RL. Recently, BID10 has proposed generative adversarial imitation learning (GAIL) on the line of those works. GAIL has achieved state-of-the art performance on a variety of continuous control tasks. However, as pointed out by BID10 , a crucial drawback of GAIL is requirement of a huge number of interactions between the learner and the environments during training 1 . Since the interactions with environment can be too much time-consuming especially in the real-world environments, we believe that model free IL could be more applicable to the real-world environments if the number could be reduced while keeping the imitation capability satisfied as well as GAIL.To reduce the number of interactions, we propose a model free, off-policy IL algorithm for continuous control. As opposed to GAIL and its variants BID2 BID30 BID8 BID12 those of which adopt a stochastic policy as the learner's policy, we adopt a deterministic policy while following adversarial training fashion as GAIL. We show that combining the deterministic policy into the adversarial off-policy IL objective derives a novel type of policy gradient which we call deterministic policy imitation gradient (DPIG). Because DPIG only integrates over the state space as deterministic policy gradient (DPG) algorithms BID25 , the number of the interactions is expected to be less than that for stochastic policy gradient (PG) which integrates over the state-action space. Besides, we introduce a function which we call state screening function (SSF) to avoid noisy policy update with states that are not typical of those appeared on the experts demonstrations.In order to evaluate our algorithm, we used 6 physics-based control tasks that were simulated with MuJoCo physics simulator BID29 . The experimental results show that our algorithm enables the learner to achieve the same performance as the expert does with at least tens of times less interactions than GAIL. It indicates that our algorithm is more applicable to the real-world environments than GAIL. A wide variety of IL methods have been proposed in these last few decades. The simplest IL method among those is BC (Pomerleau, 1991) which learns a mapping from states to actions in the expert's demonstrations using supervised learning. Since the learner with the mapping learned by BC does not interacts with the environments, inaccuracies of the mapping are never corrected once the training has done, whereas our algorithm corrects the learner's behavior through the interactions. A noticeable point in common between BC and our algorithm is that the both just consider the relationship between single time-step state-action pairs but information over entire trajectories of the behavior in the optimizations. In other words, the both assume that the reward structure is dense and the reasonable rewards for the states s ∈ S \ S E can not be defined. A drawback of BC due to ignorance of the information over the trajectories is referred to as the problem of compounding error BID21 ) -the inaccuracies compounds over time and can lead the learner to encounter unseen states in th expert's demonstrations. For the state s ∈ S β * E , it is assumed in our algorithm that the immediate reward is greater if the learner's behavior is more likely to the expert's behavior, and the expert's behavior yields the greatest cumulative reward. That is, maximizing the immediate reward for the state s ∈ S β * E ⊂ S E implies maximizing the cumulative reward for trajectories stating from the state in our algorithm, and thus the information over the trajectories is implicitly incorporated in log R ω of the objective (11). Therefore, our algorithm is less affected by the compounding error than BC.Another widely used approach for IL the combination of IRL and RL that we considered in this paper. The concept of IRL was originally proposed by BID22 , and a variety of IRL algorithms have been proposed so far. Early works on IRL BID18 BID1 BID32 represented the parameterized reward function as a linear combination of hand-designed features. Thus, its capabilities to represent the reward were limited in comparison with that of nonlinear functional representation. Indeed, applications of the early works were often limited in small discrete domains. The early works were extended to algorithms that enable to learn nonlinear functions for representing the reward BID11 BID6 . A variety of complex tasks including continuous control in the real-world environment have succeeded with the nonlinear functional representation BID6 . As well as those methods, our algorithm can utilize the nonlinear functions if it is differentiable with respect to the action.In recent years, the connection between GANs and the IL approach has been pointed out BID10 BID5 . BID10 showed that IRL is a dual problem of RL which can be deemed as a problem to match the learner's occupancy measure BID28 to that of the expert, and found a choice of regularizer for the cost function yields an objective which is analogous to that of GANs. After that, their algorithm, namely GAIL, has become a popular choice for IL and some extensions of GAIL have been proposed BID2 BID30 BID8 BID12 . However, those extensions have never addressed reducing the number of interactions during training whereas we address it, and our algorithm significantly reduce the number while keeping the imitation capability as well as GAIL.The way of deriving policy gradients using gradients of the parameterized reward function with respect to actions executed by the current learner's policy is similar to DPG BID25 and deep DPG BID13 . However, they require parameterized Q-function approximator with known reward function whereas our algorithm does not use Q-function besides the parameterized reward function learned by IRL. In IL literature, MGAIL BID2 uses the gradients derived from parameterized discriminator to update the learner's policy. However, MGAIL is modelbased method and requires to train parameterized forward-model to derive the gradients whereas our algorithm is model free. Although model based methods have been thought to need less environment interactions than model free methods in general, the experimental results showed that MGAIL needs more interactions than our model free algorithm. We think that the reasons are the need for training the forward model besides that for the policy, and lack of care for the noisy policy updates issue that MGAIL essentially has. In this paper, we proposed a model free, off-policy IL algorithm for continuous control. The experimental results showed that our algorithm enables the learner to achieve the same performance as the expert does with several tens of times less interactions than GAIL.Although we implemented shallow neural networks to represent the parameterized functions in the experiment, deep neural networks can also be applied to represent the functions in our algorithm. We expect that the that advanced techniques used in deep GANs enable us to apply our algorithm to more complex tasks.A DETAILED DESCRIPTION OF EXPERIMENT TAB2 summarizes the description of each task, an agents performance with random policy, and the performance of the experts.
We propose a model free imitation learning algorithm that is able to reduce number of interactions with environment in comparison with state-of-the-art imitation learning algorithm namely GAIL. Proposes to extend the determinist policy gradient algorithm to learn from demonstrations, while combined with a type of density estimation of the expert. This paper considers the problem of model-free imitation learning and proposes an extension of the generative adversarial imitation learning algorithm by replacing the stochastic policy of the learner with a deterministic one. The paper combines IRL, adversarial training, and ideas from deterministic policy gradients with the goal of decreasng sample complexity