A noisy and diverse demonstration set may hinder the performances of an agent aiming to acquire certain skills via imitation learning. However, state-of-the-art imitation learning algorithms often assume the optimality of the given demonstration set.
 In this paper, we address such optimal assumption by learning only from the most suitable demonstrations in a given set. Suitability of a demonstration is estimated by whether imitating it produce desirable outcomes for achieving the goals of the tasks. For more efficient demonstration suitability assessments, the learning agent should be capable of imitating a demonstration as quick as possible, which shares similar spirit with fast adaptation in the meta-learning regime. Our framework, thus built on top of Model-Agnostic Meta-Learning, evaluates how desirable the imitated outcomes are, after adaptation to each demonstration in the set. The resulting assessments hence enable us to select suitable demonstration subsets for acquiring better imitated skills. The videos related to our experiments are available at: https://sites.google.com/view/deepdj Imagine that you intend to learn how to make a free throw in basketball, which requires you to throw the ball into the basket from a fixed position. Without the proper knowledge, one may observe professional players perform a free throw on YouTube by obtaining numerous exemplary videos. However, learning from every demonstration videos might lead to worse performance, as they may contain unsuitable or even irrelevant content.The challenge of learning from noisy demonstration sets is as well crucial in the robot imitation learning regime, as demonstrations which are not aligned with achieving the intended goal deteriorate the learning process. The assumptions of optimality (or at least sub-optimality) of the demonstrations are often made in state-of-the-art imitation learning algorithms (Ross & Bagnell, 2010; Ross et al., 2011; BID10 Sermanet et al., 2018) . As a result, they are vulnerable to demonstrations that are potentially detrimental to the learning outcomes in the given set. To address assumptions of optimality, in this paper, we aim for a generic framework capable of learning from a noisy demonstration set, via evaluating the suitability of imitated skills judged by task specific heuristics.Prior works have handled deteriorated imitated outcomes due to noisy demonstration sets by utilizing expected Q-values provided in the demonstrations to avoid learning from bad demonstrated actions BID14 BID17 BID5 . However, these works require demonstrations to have a rich representation such as incorporating the aforementioned Q-values, which may neither be available in other cases nor directly applicable to the agent training environment. Our framework, on the other hand, does not require demonstrations to contain specific information, and hence is able to cope with any forms of expert demonstrations. To be specific, we propose to first assess which demonstrations in the given set might be more suitable by learning from them, and then train the agent imitating only a selected subset.In order to achieve selectively learning from suitable demonstrations, we examine if the learning outcomes are favorable after imitating each demonstration in a set. Typically in robot learning regime, we are able to receive designed feedbacks in the target training environment to evaluate how well the agent performs. Thus, in each task, we predefine specific heuristics for assessing if the agent exhibits imitation learning outcomes desirable for reaching the goals of the tasks.The key challenges for the feasibility of assessing learned outcomes from each demonstration are the efficiency and generalization ability. A framework should be capable of producing these assessable outcomes as quick as possible, and generalizing to unseen demonstrations. To this end, we propose a framework with the demonstration suitability assessor leveraging meta-learning, where we train adaptive parameters via meta-imitation-learning. The meta-imitation-learned parameters can thus: (1) produce assessable imitated outcomes at testing time quicker than both imitation learning from scratch and fine-tune a pretrained initialization BID3 , and (2) adapt to imitating newly sampled unseen demonstrations. Overall, the imitated outcomes after adaptation will be judged by task heuristics to indicate the suitability of certain demonstrations. We then train agents using imitation learning from the selected suitable demonstration subsets for obtaining better policies. We demonstrate two empirical approaches to utilize the resulting suitability assessments. One composes new subsets of demonstrations from the top ranked, the other iteratively fine-tunes the meta-learned parameters by strengthening or weakening certain demonstrations in a set according to current suitability judgments, producing a selected suitable subset at convergence.In some cases, the distribution of demonstration sets can be imbalanced or multi-modal. To prevent over-fitting to certain subsets of such demonstration distributions, we augment the meta-imitationtraining with a regularization objective-maximization of mutual information between the demonstration and the induced behavioral differences from imitating it. This additional regularization term aims to make the meta-trained parameters more responsive to the demonstrations being imitated, and as a result, help differentiate better the adapted behaviors from a noisy and diverse set.We test our framework on four different simulation sports environments in MuJuCo. Our results, both qualitative and quantitative, show that the proposed method outperforms various baselines, including vanilla MAML and fine-tuning a pretrained initialization, on learning better policies from noisy demonstration sets. We propose a framework to tackle the challenging problem -learning a good policy through imitation learning from a noisy demonstration set. Our framework, built on top of MAML with a mutual information maximized regularization, learns a set of adaptive parameters from the given noisy set. The agent should exhibit significant learning outcomes after fast adaptation to certain demonstrations where these outcomes can be evaluated via predefined task heuristics. By being a learning framework, the system learns to discover the most suitable demonstrations for the agent from the expert rather than selecting based on hand crafted judging rules. For future research direction, we hope this work can serve as the first trial to lure more advanced research on tackling imitation learning from noisy demonstration sets. For evaluating the suitability of certain demonstrations, we require a task dependent knowledge to judge from the imitation learning outcomes after the adaptation to a particular demonstration. We hereby describe the task heuristics for each environment in the following:• Free Throw: The minimum distance between the basketball and the fixed basket.• Penalty Save: The minimum distance between the agent and the incoming shot in an episode.• Handstand: The height of two feet of a humanoid if both two hands are touching the ground without letting the head to hit the ground. We accumulate this heuristics score when the aforementioned body pose condition is satisfied.• Martial Arts: The minimum distance between right foot and the target if the head of the humanoid is higher than a pelvis.Setups of our simulation environments are listed in
We propose a framework to learn a good policy through imitation learning from a noisy demonstration set via meta-training a demonstration suitability assessor. Contributes a MAML based algorithm to imitation learning which automatically determines if provided demonstrations are "suitable". A method for doing imitation learning from a set of demonstrations that includes useless behavior, which selects the useful demonstrations by their provided performance gains at the meta-training time.