Imitation Learning: A Survey of Learning Methods
---
paper_title: Towards a Simulator for Imitation Learning with Kinesthetic Bootstrapping

paper_content:

This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented.
---
paper_title: Getting Humanoids to Move and Imitate

paper_content:

A behavior-based approach to structuring and controlling complex robotic systems is described that uses imitation for interaction and learning. The use of basis behaviors, or primitives, lets humanoid robots dance the Macarena, do aerobics and throw objects.
---
paper_title: Confidence-based policy learning from demonstration using Gaussian mixture models

paper_content:

We contribute an approach for interactive policy learning through expert demonstration that allows an agent to actively request and effectively represent demonstration examples. In order to address the inherent uncertainty of human demonstration, we represent the policy as a set of Gaussian mixture models (GMMs), where each model, with multiple Gaussian components, corresponds to a single action. Incrementally received demonstration examples are used as training data for the GMM set. We then introduce our confident execution approach, which focuses learning on relevant parts of the domain by enabling the agent to identify the need for and request demonstrations for specific parts of the state space. The agent selects between demonstration and autonomous execution based on statistical analysis of the uncertainty of the learned Gaussian mixture set. As it achieves proficiency at its task and gains confidence in its actions, the agent operates with increasing autonomy, eliminating the need for unnecessary demonstrations of already acquired behavior, and reducing both the training time and the demonstration workload of the expert. We validate our approach with experiments in simulated and real robot domains.
---
paper_title: High Performance Outdoor Navigation from Overhead Data using Imitation Learning

paper_content:

A faucet includes an inner tube inserted in an upper opening formed in between a horizontal spout portion of the faucet and a downcomer spout portion having a lower tube hole of the inner tube poked into the downcomer spout portion and positioned under an overflow level of the horizontal spout portion, whereby upon a shielding or closing of a discharge outlet formed in a lowermost end of the downcomer spout portion, a water stream, as supplied by an opened faucet, will be ejected upwardly through the inner tube for rinsing or drinking purpose.
---
paper_title: Autonomous inverted helicopter flight via reinforcement learning

paper_content:

Helicopters have highly stochastic, nonlinear, dynamics, and autonomous helicopter flight is widely regarded to be a challenging control problem. As helicopters are highly unstable at low speeds, it is particularly difficult to design controllers for low speed aerobatic maneuvers. In this paper, we describe a successful application of reinforcement learning to designing a controller for sustained inverted flight on an autonomous helicopter. Using data collected from the helicopter in flight, we began by learning a stochastic, nonlinear model of the helicopter’s dynamics. Then, a reinforcement learning algorithm was applied to automatically learn a controller for autonomous inverted hovering. Finally, the resulting controller was successfully tested on our autonomous helicopter platform.
---
paper_title: Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors

paper_content:

Nonlinear dynamical systems have been used in many disciplines to model complex behaviors, including biological motor control, robotics, perception, economics, traffic prediction, and neuroscience. While often the unexpected emergent behavior of nonlinear systems is the focus of investigations, it is of equal importance to create goal-directed behavior e.g., stable locomotion from a system of coupled oscillators under perceptual guidance. Modeling goal-directed behavior with nonlinear systems is, however, rather difficult due to the parameter sensitivity of these systems, their complex phase transitions in response to subtle parameter changes, and the difficulty of analyzing and predicting their long-term behavior; intuition and time-consuming parameter tuning play a major role. This letter presents and reviews dynamical movement primitives, a line of research for modeling attractor behaviors of autonomous nonlinear dynamical systems with the help of statistical learning techniques. The essence of our approach is to start with a simple dynamical system, such as a set of linear differential equations, and transform those into a weakly nonlinear system with prescribed attractor dynamics by means of a learnable autonomous forcing term. Both point attractors and limit cycle attractors of almost arbitrary complexity can be generated. We explain the design principle of our approach and evaluate its properties in several example applications in motor control and robotics.
---
paper_title: Learning to select and generalize striking movements in robot table tennis

paper_content:

Learning new motor tasks autonomously from interaction with a human being is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. In this paper, we take the task of learning table tennis as an example and present a new framework which allows a robot to learn cooperative table tennis from interaction with a human. Therefore, the robot first learns a set of elementary table tennis hitting movements from a human teacher by kinesthetic teach-in, which is compiled into a set of dynamical system motor primitives (DMPs). Subsequently, the system generalizes these movements to a wider range of situations using our mixture of motor primitives (MoMP) approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior.
---
paper_title: A Bayesian approach to imitation learning for robot navigation

paper_content:

Driving in unknown natural outdoor terrain is a challenge for autonomous ground vehicles. It can be difficult for a robot to discern obstacles and other hazards in its environment, and characteristics of this high cost terrain may change from one environment to another, or even with different lighting conditions. One successful approach to this problem is for a robot to learn from a demonstration by a human operator. In this paper, we describe an approach to calculating terrain costs from Bayesian estimates using feature vectors measured during a short teleoperated training run in similar terrain and conditions. We describe the theory, its implementation on two different robotic systems, and results of several independently conducted field tests.
---
paper_title: Imitation learning at all levels of game-AI

paper_content:

Imitation is a powerful mechanism the human brain applies to extend its repertoire of solutions and behaviors suitable to solve problems of various kinds. From an abstract point of view, the major advantage of this strategy is that it reduces the search space of apropriate solutions. In this contribution, we discuss if and how the principle of imitation learning can facilitate the programing of life-like computer game charecters. We present different algorithms that learn from human generated training data and we show that machine learning can be applied on different levels of cognitive abstraction.
---
paper_title: Learning by demonstration with critique from a human teacher

paper_content:

Learning by demonstration can be a powerful and natural tool for developing robot control policies. That is, instead of tedious hand-coding, a robot may learn a control policy by interacting with a teacher. In this work we present an algorithm for learning by demonstration in which the teacher operates in two phases. The teacher first demonstrates the task to the learner. The teacher next critiques learner performance of the task. This critique is used by the learner to update its control policy. In our implementation we utilize a 1-Nearest Neighbor technique which incorporates both training dataset and teacher critique. Since the teacher critiques performance only, they do not need to guess at an effective critique for the underlying algorithm. We argue that this method is particularly well-suited to human teachers, who are generally better at assigning credit to performances than to algorithms. We have applied this algorithm to the simulated task of a robot intercepting a ball. Our results demonstrate improved performance with teacher critiquing, where performance is measured by both execution success and efficiency.
---
paper_title: Physical Human-Robot Interaction: Mutual Learning and Adaptation

paper_content:

Close physical interaction between robots and humans is a particularly challenging aspect of robot development. For successful interaction and cooperation, the robot must have the ability to adapt its behavior to the human counterpart. Based on our earlier work, we present and evaluate a computationally efficient machine learning algorithm that is well suited for such close-contact interaction scenarios. We show that this algorithm helps to improve the quality of the interaction between a robot and a human caregiver. To this end, we present two human-in-the-loop learning scenarios that are inspired by human parenting behavior, namely, an assisted standing-up task and an assisted walking task.
---
paper_title: Robot See, Robot Do : An Overview of Robot Imitation

paper_content:

A belt grinding machine which is equipped with an elastically flexible load beam extending over the entire operative length of an endless grinding belt for applying the necessary grinding pressure to the belt. The effective pressure-applying surface of said beam is suspended for elevational control from support elements distributed at intervals along the length of the beam and attached to an elevationally movable carrier preferably in the form of transverse beams mounted on push rods operable by a pressure medium. The push rods are associated with constant delay mechanism adjustable to the difference in time between the instant of response of a sensing element to the passage of the leading end of a workpiece and the instant the grinding action begins, on the one hand, and between the instant of response of a sensing element to the passage of the trailing end of the workpiece and the instant the grinding action terminates, on the other hand.
---
paper_title: Imitation learning through games: theory, implementation and evaluation

paper_content:

Despite a history of games-based research, academia has generally regarded ::: commercial games as a distraction from the serious business of AI, rather than as an ::: opportunity to leverage this existing domain to the advancement of our knowledge. ::: Similarly, the computer game industry still relies on techniques that were developed ::: several decades ago, and has shown little interest in adopting more progressive ::: academic approaches. In recent times, however, these attitudes have begun to change; ::: under- and post-graduate games development courses are increasingly common, ::: while the industry itself is slowly but surely beginning to recognise the potential ::: offered by modern machine-learning approaches, though games which actually ::: implement said approaches on more than a token scale remain scarce. ::: One area which has not yet received much attention from either academia or industry ::: is imitation learning, which seeks to expedite the learning process by exploiting data ::: harvested from demonstrations of a given task. While substantial work has been done ::: in developing imitation techniques for humanoid robot movement, there has been ::: very little exploration of the challenges posed by interactive computer games. Given ::: that such games generally encode reasoning and decision-making behaviours which ::: are inherently more complex and potentially more interesting than limb motion data, ::: that they often provide inbuilt facilities for recording human play, that the generation ::: and collection of training samples is therefore far easier than in robotics, and that ::: many games have vast pre-existing libraries of these recorded demonstrations, it is ::: fair to say that computer games represent an extremely fertile domain for imitation ::: learning research. ::: In this thesis, we argue in favour of using modern, commercial computer games to ::: study, model and reproduce humanlike behaviour. We provide an overview of the ::: biological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based ::: imitation. We then proceed to describe our contributions to the field of imitation ::: learning itself, which encompass three distinct categories: theory, implementation ::: and evaluation. ::: We first describe the development of a fully-featured Java API - the Quake2 Agent ::: Simulation Environment (QASE) - designed to facilitate both research and education ::: in imitation and general machine-learning, using the game Quake 2 as a testbed. We ::: outline our motivation for developing QASE, discussing the shortcomings of existing ::: APIs and the steps which we have taken to circumvent them. We describe QASE’s ::: network layer, which acts as an interface between the local AI routines and the ::: Quake 2 server on which the game environment is maintained, before detailing the ::: API’s agent architecture, which includes an interface to the MatLab programming ::: environment and the ability to parse and analyse full recordings of game sessions. ::: We conclude the chapter with a discussion of QASE’s adoption by numerous ::: universities as both an undergraduate teaching tool and research platform. ::: We then proceed to describe the various imitative mechanisms which we have ::: developed using QASE and its MatLab integration facilities. We first outline a ::: behaviour model based on a well-known psychological model of human planning. ::: Drawing upon previous research, we also identify a set of believability criteria - ::: elements of agent behaviour which are of particular importance in determining the ::: “humanness” of its in-game appearance. We then detail a reinforcement-learning ::: approach to imitating the human player’s navigation of his environment, centred ::: upon his pursuit of items as strategic goals. In the subsequent section, we describe ::: the integration of this strategic system with a Bayesian mechanism for the imitation ::: of tactical and motion-modelling behaviours. Finally, we outline a model for the ::: imitation of reactive combat behaviours; specifically, weapon-selection and aiming. Experiments are presented in each case to demonstrate the imitative mechanisms’ ::: ability to accurately reproduce observed behaviours. ::: Finally, we criticise the lack of any existing methodology to formally gauge the ::: believability of game agents, and observe that the few previous attempts have been ::: extremely ad-hoc and informal. We therefore propose a generalised approach to such ::: testing; the Bot-Oriented Turing Test (BOTT). This takes the form of an anonymous ::: online questionnaire, an accompanying protocol to which examiners should adhere, ::: and the formulation of a believability index which numerically expresses each agent’s ::: humanness as indicated by its observers, weighted by their experience and the ::: accuracy with which the agents were identified. To both validate the survey approach ::: and to determine the efficacy of our imitative models, we present a series of ::: experiments which use the believability test to evaluate our own imitation agents ::: against both human players and traditional artificial bots. We demonstrate that our ::: imitation agents perform substantially better than even a highly-regarded rule-based ::: agent, and indeed approach the believability of actual human players. ::: Some suggestions for future directions in our research, as well as a broader ::: discussion of open questions, conclude this thesis.
---
paper_title: Learning from Demonstration

paper_content:

By now it is widely accepted that learning a task from scratch, i.e., without any prior knowledge, is a daunting undertaking. Humans, however, rarely attempt to learn from scratch. They extract initial biases as well as strategies how to approach a learning problem from instructions and/or demonstrations of other humans. For teaming control, this paper investigates how learning from demonstration can be applied in the context of reinforcement learning. We consider priming the Q-function, the value function, the policy, and the model of the task dynamics as possible areas where demonstrations can speed up learning. In general nonlinear learning problems, only model-based reinforcement learning shows significant speed-up after a demonstration, while in the special case of linear quadratic regulator (LQR) problems, all methods profit from the demonstration. In an implementation of pole balancing on a complex anthropomorphic robot arm, we demonstrate that, when facing the complexities of real signal processing, model-based reinforcement learning offers the most robustness for LQR problems. Using the suggested methods, the robot learns pole balancing in just a single trial after a 30 second long demonstration of the human instructor.
---
paper_title: Computational Approaches to Motor Learning by Imitation

paper_content:

Movement imitation requires a complex set of mechanisms that map an observed movement of a teacher onto one's own movement apparatus. Relevant problems include movement recognition, pose estimation, pose tracking, body correspondence, coordinate transformation from external to egocentric space, matching of observed against previously learned movement, resolution of redundant degrees-of-freedom that are unconstrained by the observation, suitable movement representations for imitation, modularization of motor control, etc. All of these topics by themselves are active research problems in computational and neurobiological sciences, such that their combination into a complete imitation system remains a daunting undertaking-indeed, one could argue that we need to understand the complete perception-action loop. As a strategy to untangle the complexity of imitation, this paper will examine imitation purely from a computational point of view, i.e. we will review statistical and mathematical approaches that have been suggested for tackling parts of the imitation problem, and discuss their merits, disadvantages and underlying principles. Given the focus on action recognition of other contributions in this special issue, this paper will primarily emphasize the motor side of imitation, assuming that a perceptual system has already identified important features of a demonstrated movement and created their corresponding spatial information. Based on the formalization of motor control in terms of control policies and their associated performance criteria, useful taxonomies of imitation learning can be generated that clarify different approaches and future research directions.
---
paper_title: Imitation Learning of Dual-Arm Manipulation Tasks in Humanoid Robots

paper_content:

In this paper, we deal with imitation learning of arm movements in humanoid robots. Hidden Markov Models (HMM) are used to generalize movements demonstrated to a robot multiple times. They are trained with the characteristic features (key points) of each demonstration. Using the same HMM, key points that are common to all demonstrations are identified; only those are considered when reproducing a movement. We also show how HMM can be used to detect temporal dependencies between both arms in dual-arm tasks. We created a model of the human upper body to simulate the reproduction of dual-arm movements and generate natural-looking joint configurations from tracked hand paths. Results are presented and discussed.
---
paper_title: Incremental learning of gestures by imitation in a humanoid robot

paper_content:

We present an approach to teach incrementally human gestures to a humanoid robot. By using active teaching methods that puts the human teacher "in the loop" of the robot's learning, we show that the essential characteristics of a gesture can be efficiently transferred by interacting socially with the robot. In a first phase, the robot observes the user demonstrating the skill while wearing motion sensors. The motion of his/her two arms and head are recorded by the robot, projected in a latent space of motion and encoded bprobabilistically in a Gaussian Mixture Model (GMM). In a second phase, the user helps the robot refine its gesture by kinesthetic teaching, i.e. by grabbing and moving its arms throughout the movement to provide the appropriate scaffolds. To update the model of the gesture, we compare the performance of two incremental training procedures against a batch training procedure. We present experiments to show that different modalities can be combined efficiently to teach incrementally basketball officials' signals to a HOAP-3 humanoid robot.
---
paper_title: Is imitation learning the route to humanoid robots?

paper_content:

Summary This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It will be postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in form of movement primitives. It will be reviewed how research on representations of, and functional connections between action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning will also be described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical learning research. Parallels and differences between biological and computational approaches to imitation will be highlighted and an overview of current projects that actually employ imitation learning for humanoid robots will be given.
---
paper_title: Boosting Structured Prediction for Imitation Learning

paper_content:

The Maximum Margin Planning (MMP) (Ratliff et al., 2006) algorithm solves imitation learning problems by learning linear mappings from features to cost functions in a planning domain. The learned policy is the result of minimum-cost planning using these cost functions. These mappings are chosen so that example policies (or trajectories) given by a teacher appear to be lower cost (with a lossscaled margin) than any other policy for a given planning domain. We provide a novel approach, MMPBOOST , based on the functional gradient descent view of boosting (Mason et al., 1999; Friedman, 1999a) that extends MMP by “boosting” in new features. This approach uses simple binary classification or regression to improve performance of MMP imitation learning, and naturally extends to the class of structured maximum margin prediction problems. (Taskar et al., 2005) Our technique is applied to navigation and planning problems for outdoor mobile robots and robotic legged locomotion.
---
paper_title: A survey of robot learning from demonstration

paper_content:

We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research.
---
paper_title: An application of reinforcement learning to aerobatic helicopter flight

paper_content:

Autonomous helicopter flight is widely regarded to be a highly challenging control problem. This paper presents the first successful autonomous completion on a real RC helicopter of the following four aerobatic maneuvers: forward flip and sideways roll at low speed, tail-in funnel, and nose-in funnel. Our experimental results significantly extend the state of the art in autonomous helicopter flight. We used the following approach: First we had a pilot fly the helicopter to help us find a helicopter dynamics model and a reward (cost) function. Then we used a reinforcement learning (optimal control) algorithm to find a controller that is optimized for the resulting model and reward function. More specifically, we used differential dynamic programming (DDP), an extension of the linear quadratic regulator (LQR).
---
paper_title: Teaching coordinated strategies to soccer robots via imitation

paper_content:

Developing coordination among multiple agents and enabling them to exhibit teamwork is a challenging yet exciting task that can benefit many of the complex real-life problems. This research uses imitation to learn collaborative strategies for a team of agents. Imitation based learning involves learning from an expert by observing him/her demonstrating a task and then replicating it. The key idea is to involve multiple human experts during demonstration to teach autonomous agents how to work in coordination. The effectiveness of the proposed methodology has been assessed in a goal defending scenario of the RoboCup Soccer Simulation 3D league. The process involves multiple human demonstrators controlling soccer agents via game controllers and demonstrating them how to play soccer in coordination. The data gathered during this phase is used as training data to learn a classification model which is later used by the soccer agents to make autonomous decisions during actual matches. Different performance evaluation metrics are derived to compare the performance of imitating agent with that of the human-driven agent and hand-coded (if-then-else rules) agent.
---
paper_title: Computational Approaches to Motor Learning by Imitation

paper_content:

Movement imitation requires a complex set of mechanisms that map an observed movement of a teacher onto one's own movement apparatus. Relevant problems include movement recognition, pose estimation, pose tracking, body correspondence, coordinate transformation from external to egocentric space, matching of observed against previously learned movement, resolution of redundant degrees-of-freedom that are unconstrained by the observation, suitable movement representations for imitation, modularization of motor control, etc. All of these topics by themselves are active research problems in computational and neurobiological sciences, such that their combination into a complete imitation system remains a daunting undertaking-indeed, one could argue that we need to understand the complete perception-action loop. As a strategy to untangle the complexity of imitation, this paper will examine imitation purely from a computational point of view, i.e. we will review statistical and mathematical approaches that have been suggested for tackling parts of the imitation problem, and discuss their merits, disadvantages and underlying principles. Given the focus on action recognition of other contributions in this special issue, this paper will primarily emphasize the motor side of imitation, assuming that a perceptual system has already identified important features of a demonstrated movement and created their corresponding spatial information. Based on the formalization of motor control in terms of control policies and their associated performance criteria, useful taxonomies of imitation learning can be generated that clarify different approaches and future research directions.
---
paper_title: A survey of robot learning from demonstration

paper_content:

We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research.
---
paper_title: Learning Attractor Landscapes for Learning Motor Primitives

paper_content:

Many control problems take place in continuous state-action spaces, e.g., as in manipulator robotics, where the control objective is often defined as finding a desired trajectory that reaches a particular goal state. While reinforcement learning offers a theoretical framework to learn such control policies from scratch, its applicability to higher dimensional continuous state-action spaces remains rather limited to date. Instead of learning from scratch, in this paper we suggest to learn a desired complex control policy by transforming an existing simple canonical control policy. For this purpose, we represent canonical policies in terms of differential equations with well-defined attractor properties. By nonlinearly transforming the canonical attractor dynamics using techniques from nonparametric regression, almost arbitrary new nonlinear policies can be generated without losing the stability properties of the canonical system. We demonstrate our techniques in the context of learning a set of movement skills for a humanoid robot from demonstrations of a human teacher. Policies are acquired rapidly, and, due to the properties of well formulated differential equations, can be re-used and modified on-line under dynamic changes of the environment. The linear parameterization of nonparametric regression moreover lends itself to recognize and classify previously learned movement skills. Evaluations in simulations and on an actual 30 degree-of-freedom humanoid robot exemplify the feasibility and robustness of our approach.
---
paper_title: Learning Stable Nonlinear Dynamical Systems With Gaussian Mixture Models

paper_content:

This paper presents a method to learn discrete robot motions from a set of demonstrations. We model a motion as a nonlinear autonomous (i.e., time-invariant) dynamical system (DS) and define sufficient conditions to ensure global asymptotic stability at the target. We propose a learning method, which is called Stable Estimator of Dynamical Systems (SEDS), to learn the parameters of the DS to ensure that all motions closely follow the demonstrations while ultimately reaching and stopping at the target. Time-invariance and global asymptotic stability at the target ensures that the system can respond immediately and appropriately to perturbations that are encountered during the motion. The method is evaluated through a set of robot experiments and on a library of human handwriting motions.
---
paper_title: Motion Imitation Based on Sparsely Sampled Correspondence

paper_content:

Existing techniques for motion imitation often suffer a certain level of latency due to their computational overhead or a large set of correspondence samples to search. To achieve real-time imitation with small latency, we present a framework in this paper to reconstruct motion on humanoids based on sparsely sampled correspondence. The imitation problem is formulated as finding the projection of a point from the configuration space of a human's poses into the configuration space of a humanoid. An optimal projection is defined as the one that minimizes a back-projected deviation among a group of candidates, which can be determined in a very efficient way. Benefited from this formulation, effective projections can be obtained by using sparse correspondence. Methods for generating these sparse correspondence samples have also been introduced. Our method is evaluated by applying the human's motion captured by a RGB-D sensor to a humanoid in real-time. Continuous motion can be realized and used in the example application of tele-operation.
---
paper_title: Robot Learning from Failed Demonstrations

paper_content:

Robot Learning from Demonstration (RLfD) seeks to enable lay users to encode desired robot behaviors as autonomous controllers. Current work uses a human’s demonstration of the target task to initialize the robot’s policy, and then improves its performance either through practice (with a known reward function), or additional human interaction. In this article, we focus on the initialization step and consider what can be learned when the humans do not provide successful examples. We develop probabilistic approaches that avoid reproducing observed failures while leveraging the variance across multiple attempts to drive exploration. Our experiments indicate that failure data do contain information that can be used to discover successful means to accomplish tasks. However, in higher dimensions, additional information from the user will most likely be necessary to enable efficient failure-based learning.
---
paper_title: A Behavior Generation Framework for Robots to Learn from Demonstrations

paper_content:

This paper proposes a framework of generating behavior sequences for robots, especially for humanoid robots, to perform complex tasks. This framework provides a method for the robot to generalize common features of demonstrated behaviors, to store the learned behaviors in the memory system, to construct a behavior graph to describe relationships among learned behaviors, to find and assemble a behavior sequence, and to generate similar motion trajectories of basic behaviors when it is placed in a similar but slightly different task-relevant situation. Additionally, we successfully use behavior graph to describe the dynamic relationship among behaviors and we successfully apply shortest path searching methods in behavior sequence generation, which provides a novel solution to associate knowledge representation with behavior generation. Simulation and experiments are carried on a humanoid robot to validate our proposed framework.
---
paper_title: Learning from Limited Demonstrations

paper_content:

We propose a Learning from Demonstration (LfD) algorithm which leverages expert data, even if they are very few or inaccurate. We achieve this by using both expert data, as well as reinforcement signals gathered through trial-and-error interactions with the environment. The key idea of our approach, Approximate Policy Iteration with Demonstration (APID), is that expert's suggestions are used to define linear constraints which guide the optimization performed by Approximate Policy Iteration. We prove an upper bound on the Bellman error of the estimate computed by APID at each iteration. Moreover, we show empirically that APID outperforms pure Approximate Policy Iteration, a state-of-the-art LfD algorithm, and supervised learning in a variety of scenarios, including when very few and/or suboptimal demonstrations are available. Our experiments include simulations as well as a real robot path-finding task.
---
paper_title: A humanoid robot standing up through learning from demonstration using a multimodal reward function

paper_content:

Humans are known to manage postural movements in a very elegant manner. In the task of standing up from a chair, a humanoid robot can benefit from the variability of human demonstrations. In this paper we propose a novel method for humanoid robots to imitate a dynamic postural movement demonstrated by humans. Since the kinematics of human participants and the humanoid robot used in this experiment are different, we solve the correspondence problem by making comparisons in a common reward space defined by a multimodal reward function composed of balance and effort terms. We fitted a fully actuated triple inverted pendulum to model both human and robot. We used Differential Evolution to find the optimal articular trajectory that minimizes the Kullback-Leibler difference between the human's and robot's reward profile subject to constraints.
---
paper_title: Humanoid Robot's Visual Imitation of 3-D Motion of a Human Subject Using Neural-Network-Based Inverse Kinematics

paper_content:

A sequence of 3-D motion images of a human facing an imitator [i.e., a humanoid robot (HR)] is captured using a stereo-vision system installed on the HR. After acquiring enough motion sequences via image ratio and background registration, the estimation of seven feature points (i.e., the head, two arm tips, two leg tips, and two elbows) of the human are obtained. Using five feature points (i.e., the head, two arm tips, and two leg tips), the extraction of key posture frames from the recorded videos is achieved. To ensure a stable imitation motion of the HR, the human motions of the lower body (LB) and the upper body (UB) are imitated separately. All 11 stable motions for the LB are classified by the motion direction, the motion state, and the support phase, whereas only the two arm tips are employed to imitate the motions of the UB. Due to the disadvantages of traditional inverse kinematics (IK), neural-network-based IK is developed to reduce the computation time and improve the accuracy of the time response of the system. Finally, the combined motions of the LB and the UB with suitable interpolation and time intervals are applied to imitate the 3-D motion of a human. The corresponding experimental results confirm the efficacy of the proposed technique.
---
paper_title: Imitating human playing styles in Super Mario Bros

paper_content:

Abstract We describe and compare several methods for generating game character controllers that mimic the playing style of a particular human player, or of a population of human players, across video game levels. Similarity in playing style is measured through an evaluation framework, that compares the play trace of one or several human players with the punctuated play trace of an AI player. The methods that are compared are either hand-coded, direct (based on supervised learning) or indirect (based on maximising a similarity measure). We find that a method based on neuroevolution performs best both in terms of the instrumental similarity measure and in phenomenological evaluation by human spectators. A version of the classic platform game “Super Mario Bros” is used as the testbed game in this study but the methods are applicable to other games that are based on character movement in space.
---
paper_title: Learning human arm movements by imitation: : Evaluation of a biologically inspired connectionist architecture

paper_content:

Abstract This paper evaluates a model of human imitation of abstract, two-arm movements. The model consists of a hierarchy of artificial neural networks, which are abstractions of brain regions involved in visuo-motor control. The model is validated in a biomechanical simulation of a 37 degree of freedom (DOF) humanoid. Input to the model are data from human arm movements recorded using video and marker-based tracking systems. Results show a high qualitative and quantitative agreement with human data. The model’s reproduction is better or comparable to that of human subjects imitating the same movements.
---
paper_title: Is imitation learning the route to humanoid robots?

paper_content:

Summary This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It will be postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in form of movement primitives. It will be reviewed how research on representations of, and functional connections between action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning will also be described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical learning research. Parallels and differences between biological and computational approaches to imitation will be highlighted and an overview of current projects that actually employ imitation learning for humanoid robots will be given.
---
paper_title: From motor babbling to hierarchical learning by imitation: A robot developmental pathway

paper_content:

How does an individual use the knowledge ::: acquired through self exploration as a manipulable model through which to understand ::: others and benefit from their knowledge? ::: How can developmental and social learning be ::: combined for their mutual benefit? In this ::: paper we review a hierarchical architecture ::: (HAMMER) which allows a principled way ::: for combining knowledge through exploration ::: and knowledge from others, through the creation and use of multiple inverse and forward ::: models. We describe how Bayesian Belief Networks can be used to learn the association ::: between a robot’s motor commands and sensory consequences (forward models), and how ::: the inverse association can be used for imitation. Inverse models created through self ::: exploration, as well as those from observing ::: others can coexist and compete in a principled unified framework, that utilises the simulation theory of mind approach to mentally ::: rehearse and understand the actions of others.
---
paper_title: Humanoid Robot's Visual Imitation of 3-D Motion of a Human Subject Using Neural-Network-Based Inverse Kinematics

paper_content:

A sequence of 3-D motion images of a human facing an imitator [i.e., a humanoid robot (HR)] is captured using a stereo-vision system installed on the HR. After acquiring enough motion sequences via image ratio and background registration, the estimation of seven feature points (i.e., the head, two arm tips, two leg tips, and two elbows) of the human are obtained. Using five feature points (i.e., the head, two arm tips, and two leg tips), the extraction of key posture frames from the recorded videos is achieved. To ensure a stable imitation motion of the HR, the human motions of the lower body (LB) and the upper body (UB) are imitated separately. All 11 stable motions for the LB are classified by the motion direction, the motion state, and the support phase, whereas only the two arm tips are employed to imitate the motions of the UB. Due to the disadvantages of traditional inverse kinematics (IK), neural-network-based IK is developed to reduce the computation time and improve the accuracy of the time response of the system. Finally, the combined motions of the LB and the UB with suitable interpolation and time intervals are applied to imitate the 3-D motion of a human. The corresponding experimental results confirm the efficacy of the proposed technique.
---
paper_title: Using advice to transfer knowledge acquired in one reinforcement learning task to another

paper_content:

We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice.
---
paper_title: Multi-column Deep Neural Networks for Image Classification

paper_content:

Traditional methods of computer vision and machine learning cannot match human performance on tasks such as the recognition of handwritten digits or traffic signs. Our biologically plausible, wide and deep artificial neural network architectures can. Small (often minimal) receptive fields of convolutional winner-take-all neurons yield large network depth, resulting in roughly as many sparsely connected neural layers as found in mammals between retina and visual cortex. Only winner neurons are trained. Several deep neural columns become experts on inputs preprocessed in different ways; their predictions are averaged. Graphics cards allow for fast training. On the very competitive MNIST handwriting benchmark, our method is the first to achieve near-human performance. On a traffic sign recognition benchmark it outperforms humans by a factor of two. We also improve the state-of-the-art on a plethora of common image classification benchmarks.
---
paper_title: What is the Teacher’s Role in Robot Programming by Demonstration? Toward Benchmarks for Improved Learning

paper_content:

Robot programming by demonstration (RPD) covers methods by which a robot learns new skills through human guidance. We present an interactive, multimodal RPD framework using active teaching methods that places the human teacher in the robot's learning loop. Two experiments are presented in which observational learning is first used to demonstrate a manipulation skill to a HOAP-3 humanoid robot by using motion sensors attached to the teacher's body. Then, putting the robot through the motion, the teacher incrementally refines the robot's skill by moving its arms manually, providing the appropriate scaffolds to reproduce the action. An incremental teaching scenario is proposed based on insights from various fields addressing developmental, psychological, and social issues related to teaching mechanisms in humans. Based on this analysis, different benchmarks are suggested to evaluate the setup further.
---
paper_title: Towards a Simulator for Imitation Learning with Kinesthetic Bootstrapping

paper_content:

This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented.
---
paper_title: Asynchronous Methods for Deep Reinforcement Learning

paper_content:

We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
---
paper_title: Imagenet classification with deep convolutional neural networks

paper_content:

We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.
---
paper_title: Continuous control with deep reinforcement learning

paper_content:

We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.
---
paper_title: The Arcade Learning Environment: An Evaluation Platform for General Agents

paper_content:

In this extended abstract we introduce the Arcade Learning Environment (ALE): both a challenge problem and a platform and methodology for evaluating the development of general, domain-independent AI technology. ALE provides an interface to hundreds of Atari 2600 game environments, each one different, interesting, and designed to be a challenge for human players. ALE presents significant research challenges for reinforcement learning, model learning, model-based planning, imitation learning, transfer learning, and intrinsic motivation. Most importantly, it provides a rigorous testbed for evaluating and comparing approaches to these problems. We illustrate the promise of ALE by presenting a benchmark set of domain-independent agents designed using well-established AI techniques for both reinforcement learning and planning. In doing so, we also propose an evaluation methodology made possible by ALE, reporting empirical results on over 55 different games. We conclude with a brief update on the latest ALE developments. All of the software, including the benchmark agents, is publicly available.
---
paper_title: Using PCA to Efficiently Represent State Spaces

paper_content:

Reinforcement learning algorithms need to deal with the exponential growth of states and actions when exploring optimal control in high-dimensional spaces. This is known as the curse of dimensionality. By projecting the agent's state onto a low-dimensional manifold, we can represent the state space in a smaller and more efficient representation. By using this representation during learning, the agent can converge to a good policy much faster. We test this approach in the Mario Benchmarking Domain. When using dimensionality reduction in Mario, learning converges much faster to a good policy. But, there is a critical convergence-performance trade-off. By projecting onto a low-dimensional manifold, we are ignoring important data. In this paper, we explore this trade-off of convergence and performance. We find that learning in as few as 4 dimensions (instead of 9), we can improve performance past learning in the full dimensional space at a faster convergence rate.
---
paper_title: Physical Human-Robot Interaction: Mutual Learning and Adaptation

paper_content:

Close physical interaction between robots and humans is a particularly challenging aspect of robot development. For successful interaction and cooperation, the robot must have the ability to adapt its behavior to the human counterpart. Based on our earlier work, we present and evaluate a computationally efficient machine learning algorithm that is well suited for such close-contact interaction scenarios. We show that this algorithm helps to improve the quality of the interaction between a robot and a human caregiver. To this end, we present two human-in-the-loop learning scenarios that are inspired by human parenting behavior, namely, an assisted standing-up task and an assisted walking task.
---
paper_title: Learning Two-Person Interaction Models for Responsive Synthetic Humanoids

paper_content:

JVRB, 11(2014), no. 1. - Imitation learning is a promising approach for generating life-like behaviors of virtual humans and humanoid robots. So far, however, imitation learning has been mostly restricted to single agent settings where observed motions are adapted to new environment conditions but not to the dynamic behavior of interaction partners. In this paper, we introduce a new imitation learning approach that is based on the simultaneous motion capture of two human interaction partners. From the observed interactions, low-dimensional motion models are extracted and a mapping between these motion models is learned. This interaction model allows the real-time generation of agent behaviors that are responsive to the body movements of an interaction partner. The interaction model can be applied both to the animation of virtual characters as well as to the behavior generation for humanoid robots.
---
paper_title: Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning

paper_content:

The combination of modern Reinforcement Learning and Deep Learning approaches holds the promise of making significant progress on challenging applications requiring both rich perception and policy-selection. The Arcade Learning Environment (ALE) provides a set of Atari games that represent a useful benchmark set of such applications. A recent breakthrough in combining model-free reinforcement learning with deep learning, called DQN, achieves the best real-time agents thus far. Planning-based approaches achieve far higher scores than the best model-free approaches, but they exploit information that is not available to human players, and they are orders of magnitude slower than needed for real-time play. Our main goal in this work is to build a better real-time Atari game playing agent than DQN. The central idea is to use the slow planning-based agents to provide training data for a deep-learning architecture capable of real-time play. We proposed new agents based on this idea and show that they outperform DQN.
---
paper_title: Human-level control through deep reinforcement learning

paper_content:

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
---
paper_title: Evolving large-scale neural networks for vision-based reinforcement learning

paper_content:

The idea of using evolutionary computation to train artificial neural networks, or neuroevolution (NE), for reinforcement learning (RL) tasks has now been around for over 20 years. However, as RL tasks become more challenging, the networks required become larger, as do their genomes. But, scaling NE to large nets (i.e. tens of thousands of weights) is infeasible using direct encodings that map genes one-to-one to network components. In this paper, we scale-up our compressed network encoding where network weight matrices are represented indirectly as a set of Fourier-type coefficients, to tasks that require very-large networks due to the high-dimensionality of their input space. The approach is demonstrated successfully on two reinforcement learning tasks in which the control networks receive visual input: (1) a vision-based version of the octopus control task requiring networks with over 3 thousand weights, and (2) a version of the TORCS driving game where networks with over 1 million weights are evolved to drive a car around a track using video images from the driver's perspective.
---
paper_title: Confidence-based policy learning from demonstration using Gaussian mixture models

paper_content:

We contribute an approach for interactive policy learning through expert demonstration that allows an agent to actively request and effectively represent demonstration examples. In order to address the inherent uncertainty of human demonstration, we represent the policy as a set of Gaussian mixture models (GMMs), where each model, with multiple Gaussian components, corresponds to a single action. Incrementally received demonstration examples are used as training data for the GMM set. We then introduce our confident execution approach, which focuses learning on relevant parts of the domain by enabling the agent to identify the need for and request demonstrations for specific parts of the state space. The agent selects between demonstration and autonomous execution based on statistical analysis of the uncertainty of the learned Gaussian mixture set. As it achieves proficiency at its task and gains confidence in its actions, the agent operates with increasing autonomy, eliminating the need for unnecessary demonstrations of already acquired behavior, and reducing both the training time and the demonstration workload of the expert. We validate our approach with experiments in simulated and real robot domains.
---
paper_title: Learning Manipulation Trajectories Using Recurrent Neural Networks

paper_content:

Robots assisting disabled or elderly people in the performance of activities of daily living need to perform complex manipulation tasks which are highly dependent on the environment and preferences of the user. In addition, these environments and users are not suitable for the collection of massive amounts of training data, as the manipulated objects can be fragile, and the wheelchair-bound users might have difficulty recovering from a failed manipulation task. ::: In this paper, we propose an end-to-end learning mechanism for the type of complex robot arm trajectories used in manipulation tasks for assistive robots. The trajectory is learned using a recurrent neural network that can generate the trajectory in real-time based on the current situation of the end-effector, the objects in the environment and the preferences of the user. The learning data is acquired from a simulation environment where the human can demonstrate the task in a simulation closely modeling his or her own environment. Experiments using two different manipulation tasks show that the robot can learn the manipulation planning as well the ability to recover from failure.
---
paper_title: Boosting Structured Prediction for Imitation Learning

paper_content:

The Maximum Margin Planning (MMP) (Ratliff et al., 2006) algorithm solves imitation learning problems by learning linear mappings from features to cost functions in a planning domain. The learned policy is the result of minimum-cost planning using these cost functions. These mappings are chosen so that example policies (or trajectories) given by a teacher appear to be lower cost (with a lossscaled margin) than any other policy for a given planning domain. We provide a novel approach, MMPBOOST , based on the functional gradient descent view of boosting (Mason et al., 1999; Friedman, 1999a) that extends MMP by “boosting” in new features. This approach uses simple binary classification or regression to improve performance of MMP imitation learning, and naturally extends to the class of structured maximum margin prediction problems. (Taskar et al., 2005) Our technique is applied to navigation and planning problems for outdoor mobile robots and robotic legged locomotion.
---
paper_title: A survey of robot learning from demonstration

paper_content:

We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research.
---
paper_title: Teaching coordinated strategies to soccer robots via imitation

paper_content:

Developing coordination among multiple agents and enabling them to exhibit teamwork is a challenging yet exciting task that can benefit many of the complex real-life problems. This research uses imitation to learn collaborative strategies for a team of agents. Imitation based learning involves learning from an expert by observing him/her demonstrating a task and then replicating it. The key idea is to involve multiple human experts during demonstration to teach autonomous agents how to work in coordination. The effectiveness of the proposed methodology has been assessed in a goal defending scenario of the RoboCup Soccer Simulation 3D league. The process involves multiple human demonstrators controlling soccer agents via game controllers and demonstrating them how to play soccer in coordination. The data gathered during this phase is used as training data to learn a classification model which is later used by the soccer agents to make autonomous decisions during actual matches. Different performance evaluation metrics are derived to compare the performance of imitating agent with that of the human-driven agent and hand-coded (if-then-else rules) agent.
---
paper_title: Learning motor primitives for robotics

paper_content:

The acquisition and self-improvement of novel motor skills is among the most important problems in robotics. Motor primitives offer one of the most promising frameworks for the application of machine learning techniques in this context. Employing an improved form of the dynamic systems motor primitives originally introduced by Ijspeert et al. [2], we show how both discrete and rhythmic tasks can be learned using a concerted approach of both imitation and reinforcement learning. For doing so, we present both learning algorithms and representations targeted for the practical application in robotics. Furthermore, we show that it is possible to include a start-up phase in rhythmic primitives. We show that two new motor skills, i.e., Ball-in-a-Cup and Ball-Paddling, can be learned on a real Barrett WAM robot arm at a pace similar to human learning while achieving a significantly more reliable final performance.
---
paper_title: Learning to select and generalize striking movements in robot table tennis

paper_content:

Learning new motor tasks autonomously from interaction with a human being is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. In this paper, we take the task of learning table tennis as an example and present a new framework which allows a robot to learn cooperative table tennis from interaction with a human. Therefore, the robot first learns a set of elementary table tennis hitting movements from a human teacher by kinesthetic teach-in, which is compiled into a set of dynamical system motor primitives (DMPs). Subsequently, the system generalizes these movements to a wider range of situations using our mixture of motor primitives (MoMP) approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior.
---
paper_title: Policy Search for Motor Primitives in Robotics

paper_content:

Many motor skills in humanoid robotics can be learned using parametrized motor primitives as done in imitation learning. However, most interesting motor learning problems are high-dimensional reinforcement learning problems often beyond the reach of current methods. In this paper, we extend previous work on policy learning from the immediate reward case to episodic reinforcement learning. We show that this results in a general, common framework also connected to policy gradient methods and yielding a novel algorithm for policy learning that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable to complex motor learning tasks. We compare this algorithm to several well-known parametrized policy search methods and show that it outperforms them. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task using a real Barrett WAM™ robot arm.
---
paper_title: Learning rhythmic movements by demonstration using nonlinear oscillators

paper_content:

ATR Human Information Science Laboratories, Kyoto, JapanEmail: ijspeert@usc.edu, jun@his.atr.co.jp, sschaal@usc.eduAbstractThis paper presents a new approach to the generationof rhythmic movement patterns with nonlinear dy-namical systems. Starting from a canonical limit cy-cle oscillator with well-de ned stability properties, wemodify the attractor landscape of the canonical sys-temby meansof statisticallearning methods toembedarbitrary smooth target patterns, however, withoutlosing the stability properties of the canonical sys-tem. In contrast to non-autonomous movement rep-resentations like splines, the learned pattern gener-ators remain autonomous dynamical systems whichrobustly cope with external perturbations that disruptthe time ow of the original pattern, and which canalso be modi ed on-line by additional perceptual vari-ables. A simple extension allows to cope with mul-tiple degrees-of-freedom (DOF) patterns, where allDOFs share the same fundamental frequency but,otherwise, can move in arbitrary phase and ampli-tude o sets to each other. We evaluate our meth-ods in learning from demonstration with an actual30 DOF humanoid robot. Figure-8 and drummingmovements are demonstrated by a human, recordedin joint angle space with an exoskeleton, and em-bedded in multi-dimensional rhythmic pattern gener-ators. The learned patterns can be used by the robotin various workspace locations and from arbitraryinitial conditions. Spatial and temporal invarianceof the pattern generators allow easy amplitude andspeed scaling without losing the qualitative signatureof a movement. This novel way of creating rhyth-mic patterns could tremendously facilitate rhythmicmovement generation, in particular in locomotion ofrobots and neural prosthetics in clinical applications.
---
paper_title: Efficient movement representation by embedding Dynamic Movement Primitives in deep autoencoders

paper_content:

Predictive modeling of human or humanoid movement becomes increasingly complex as the dimensionality of those movements grows. Dynamic Movement Primitives (DMP) have been shown to be a powerful method of representing such movements, but do not generalize well when used in configuration or task space. To solve this problem we propose a model called autoencoded dynamic movement primitive (AE-DMP) which uses deep autoencoders to find a representation of movement in a latent feature space, in which DMP can optimally generalize. The architecture embeds DMP into such an autoencoder and allows the whole to be trained as a unit. To further improve the model for multiple movements, sparsity is added for the feature layer neurons; therefore, various movements can be observed clearly in the feature space. After training, the model finds a single hidden neuron from the sparsity that can efficiently generate new movements. Our experiments clearly demonstrate the efficiency of missing data imputation using 50-dimensional human movement data.
---
paper_title: Statistical dynamical systems for skills acquisition in humanoids

paper_content:

Learning by imitation in humanoids is challenging due to the unpredictable environments these robots have to face during reproduction. Two sets of tools are relevant for this purpose: 1) probabilistic machine learning methods that can extract and exploit the regularities and important features of the task; and 2) dynamical systems that can cope with perturbation in real-time without having to replan the whole movement. We present a learning by imitation approach combining the two benefits. It is based on a superposition of virtual spring-damper systems to drive a humanoid robot's movement. The method relies on a statistical description of the springs attractor points acting in different candidate frames of reference. It extends dynamic movement primitives models by formulating the dynamical systems parameters estimation problem as a Gaussian mixture regression problem with projection in different coordinate systems. The robot exploits local variability information extracted from multiple demonstrations of movements to determine which frames are relevant for the task, and how the movement should be modulated with respect to these frames. The approach is tested on the new prototype of the COMAN compliant humanoid with time-based and time-invariant movements, including bimanual coordination skills.
---
paper_title: A.J.: Dynamics systems vs. optimal control — a unifying view

paper_content:

In the past, computational motor control has been approached from at least two major frameworks: the dynamic systems approach and the viewpoint of optimal control. The dynamic system approach emphasizes motor control as a process of self-organization between an animal and its environment. Nonlinear differential equations that can model entrainment and synchronization behavior are among the most favorable tools of dynamic systems modelers. In contrast, optimal control approaches view motor control as the evolutionary or development result of a nervous system that tries to optimize rather general organizational principles, e.g., energy consumption or accurate task achievement. Optimal control theory is usually employed to develop appropriate theories. Interestingly, there is rather little interaction between dynamic systems and optimal control modelers as the two approaches follow rather different philosophies and are often viewed as diametrically opposing. In this paper, we develop a computational approach to motor control that offers a unifying modeling framework for both dynamic systems and optimal control approaches. In discussions of several behavioral experiments and some theoretical and robotics studies, we demonstrate how our computational ideas allow both the representation of self-organizing processes and the optimization of movement based on reward criteria. Our modeling framework is rather simple and general, and opens opportunities to revisit many previous modeling results from this novel unifying view.
---
paper_title: Learning perceptual coupling for motor primitives

paper_content:

Dynamic system-based motor primitives have enabled robots to learn complex tasks ranging from Tennis-swings to locomotion. However, to date there have been only few extensions which have incorporated perceptual coupling to variables of external focus, and, furthermore, these modifications have relied upon handcrafted solutions. Humans learn how to couple their movement primitives with external variables. Clearly, such a solution is needed in robotics. In this paper, we propose an augmented version of the dynamic systems motor primitives which incorporates perceptual coupling to an external variable. The resulting perceptually driven motor primitives include the previous primitives as a special case and can inherit some of their interesting properties. We show that these motor primitives can perform complex tasks such a Ball-in-a-Cup or Kendama task even with large variances in the initial conditions where a skilled human player would be challenged. For doing so, we initialize the motor primitives in the traditional way by imitation learning without perceptual coupling. Subsequently, we improve the motor primitives using a novel reinforcement learning method which is particularly well-suited for motor primitives.
---
paper_title: Learning from demonstration and adaptation of biped locomotion

paper_content:

In this paper, we introduce a framework for learning biped locomotion using dynamical movement primitives based on non-linear oscillators. Our ultimate goal is to establish a design principle of a controller in order to achieve natural human-like locomotion. We suggest dynamical movement primitives as a central pattern generator (CPG) of a biped robot, an approach we have previously proposed for learning and encoding complex human movements. Demonstrated trajectories are learned through movement primitives by locally weighted regression, and the frequency of the learned trajectories is adjusted automatically by a novel frequency adaptation algorithm based on phase resetting and entrainment of coupled oscillators. Numerical simulations and experimental implementation on a physical robot demonstrate the effectiveness of the proposed locomotion controller. © 2004 Elsevier B.V. All rights reserved.
---
paper_title: Learning optimal controllers in human-robot cooperative transportation tasks with position and force constraints

paper_content:

Human-robot collaboration seeks to have humans and robots closely interacting in everyday situations. For some tasks, physical contact between the user and the robot may occur, originating significant challenges at safety, cognition, perception and control levels, among others. This paper focuses on robot motion adaptation to parameters of a collaborative task, extraction of the desired robot behavior, and variable impedance control for human-safe interaction. We propose to teach a robot cooperative behaviors from demonstrations, which are probabilistically encoded by a task-parametrized formulation of a Gaussian mixture model. Such encoding is later used for specifying both the desired state of the robot, and an optimal feedback control law that exploits the variability in position, velocity and force spaces observed during the demonstrations. The whole framework allows the robot to modify its movements as a function of parameters of the task, while showing different impedance behaviors. Tests were successfully carried out in a scenario where a 7 DOF backdrivable manipulator learns to cooperate with a human to transport an object.
---
paper_title: Learning Attractor Landscapes for Learning Motor Primitives

paper_content:

Many control problems take place in continuous state-action spaces, e.g., as in manipulator robotics, where the control objective is often defined as finding a desired trajectory that reaches a particular goal state. While reinforcement learning offers a theoretical framework to learn such control policies from scratch, its applicability to higher dimensional continuous state-action spaces remains rather limited to date. Instead of learning from scratch, in this paper we suggest to learn a desired complex control policy by transforming an existing simple canonical control policy. For this purpose, we represent canonical policies in terms of differential equations with well-defined attractor properties. By nonlinearly transforming the canonical attractor dynamics using techniques from nonparametric regression, almost arbitrary new nonlinear policies can be generated without losing the stability properties of the canonical system. We demonstrate our techniques in the context of learning a set of movement skills for a humanoid robot from demonstrations of a human teacher. Policies are acquired rapidly, and, due to the properties of well formulated differential equations, can be re-used and modified on-line under dynamic changes of the environment. The linear parameterization of nonparametric regression moreover lends itself to recognize and classify previously learned movement skills. Evaluations in simulations and on an actual 30 degree-of-freedom humanoid robot exemplify the feasibility and robustness of our approach.
---
paper_title: Teaching collaborative multi-robot tasks through demonstration

paper_content:

Humanoid robots working alongside humans in everyday environments is a long standing goal of the robotics community. To achieve this goal, methods for developing new robot behaviors that are intuitive and accessible to non-programmers are required. In this paper, we present a demonstration-based method for teaching distributed autonomous robots to coordinate their actions and perform collaborative multi-robot tasks. Within the presented framework, each robot learns an individual policy from teacher demonstrations using a confidence-based algorithm. Based on this learning approach, we contribute three techniques for teaching multi-robot coordination using different information sharing strategies. We evaluate and compare these approaches by teaching two Sony QRIO humanoid robots to perform three collaborative ball sorting tasks.
---
paper_title: Learning tasks from observation and practice

paper_content:

Abstract This paper presents a framework that gives robots the ability to initially learn a task behavior from observing others. The framework includes a method for the robots to increase performance while operating in the task environment. We demonstrate this approach applied to air hockey and the marble maze task. Our robots initially learn to perform the tasks using learning from observation, and then increase their performance through practice.
---
paper_title: Towards a Simulator for Imitation Learning with Kinesthetic Bootstrapping

paper_content:

This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented.
---
paper_title: Towards automatic personalised content creation for racing games

paper_content:

Evolutionary algorithms are commonly used to create high-performing strategies or agents for computer games. In this paper, we instead choose to evolve the racing tracks in a car racing game. An evolvable track representation is devised, and a multiobjective evolutionary algorithm maximises the entertainment value of the track relative to a particular human player. This requires a way to create accurate models of players' driving styles, as well as a tentative definition of when a racing track is fun, both of which are provided. We believe this approach opens up interesting new research questions and is potentially applicable to commercial racing games.
---
paper_title: A survey of robot learning from demonstration

paper_content:

We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research.
---
paper_title: Programming robots using reinforcement learning and teaching

paper_content:

Programming robots is a tedious task. So, there is growing interest in building robots which can learn by themselves. Self-improving, which involves trial and error, however, is often a slow process and could be hazardous in a hostile environment. By teaching robots how tasks can be achieved, learning time can be shortened and hazard can be minimized. This paper presents a general approach to making robots which can improve their performance from experiences as well as from being taught. Based on this proposed approach and other learning speedup techniques, a simulated learning robot was developed and could learn three moderately complex behaviors, which were then integrated in a subsumption style so that the robot could navigate and recharge itself. Interestingly, a real robot could actually use what was learned in the simulator to operate in the real world quite successfully.
---
paper_title: Reinforcement learning from demonstration through shaping

paper_content:

Reinforcement learning describes how a learning agent can achieve optimal behaviour based on interactions with its environment and reward feedback. A limiting factor in reinforcement learning as employed in artificial intelligence is the need for an often prohibitively large number of environment samples before the agent reaches a desirable level of performance. Learning from demonstration is an approach that provides the agent with demonstrations by a supposed expert, from which it should derive suitable behaviour. Yet, one of the challenges of learning from demonstration is that no guarantees can be provided for the quality of the demonstrations, and thus the learned behavior. In this paper, we investigate the intersection of these two approaches, leveraging the theoretical guarantees provided by reinforcement learning, and using expert demonstrations to speed up this learning by biasing exploration through a process called reward shaping. This approach allows us to leverage human input without making an erroneous assumption regarding demonstration optimality. We show experimentally that this approach requires significantly fewer demonstrations, is more robust against suboptimality of demonstrations, and achieves much faster learning than the recently developed HAT algorithm.
---
paper_title: Self-improving reactive agents based on reinforcement learning, planning and teaching

paper_content:

To date, reinforcement learning has mostly been studied solving simple learning tasks. Reinforcement learning methods that have been studied so far typically converge slowly. The purpose of this work is thus two-fold: 1) to investigate the utility of reinforcement learning in solving much more complicated learning tasks than previously studied, and 2) to investigate methods that will speed up reinforcement learning. ::: ::: This paper compares eight reinforcement learning frameworks: adaptive heuristic critic (AHC) learning due to Sutton, Q-learning due to Watkins, and three extensions to both basic methods for speeding up learning. The three extensions are experience replay, learning action models for planning, and teaching. The frameworks were investigated using connectionism as an approach to generalization. To evaluate the performance of different frameworks, a dynamic environment was used as a testbed. The environment is moderately complex and nondeterministic. This paper describes these frameworks and algorithms in detail and presents empirical evaluation of the frameworks.
---
paper_title: Learning to select and generalize striking movements in robot table tennis

paper_content:

Learning new motor tasks autonomously from interaction with a human being is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. In this paper, we take the task of learning table tennis as an example and present a new framework which allows a robot to learn cooperative table tennis from interaction with a human. Therefore, the robot first learns a set of elementary table tennis hitting movements from a human teacher by kinesthetic teach-in, which is compiled into a set of dynamical system motor primitives (DMPs). Subsequently, the system generalizes these movements to a wider range of situations using our mixture of motor primitives (MoMP) approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior.
---
paper_title: Apprenticeship learning via inverse reinforcement learning

paper_content:

We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function.
---
paper_title: Learning rhythmic movements by demonstration using nonlinear oscillators

paper_content:

ATR Human Information Science Laboratories, Kyoto, JapanEmail: ijspeert@usc.edu, jun@his.atr.co.jp, sschaal@usc.eduAbstractThis paper presents a new approach to the generationof rhythmic movement patterns with nonlinear dy-namical systems. Starting from a canonical limit cy-cle oscillator with well-de ned stability properties, wemodify the attractor landscape of the canonical sys-temby meansof statisticallearning methods toembedarbitrary smooth target patterns, however, withoutlosing the stability properties of the canonical sys-tem. In contrast to non-autonomous movement rep-resentations like splines, the learned pattern gener-ators remain autonomous dynamical systems whichrobustly cope with external perturbations that disruptthe time ow of the original pattern, and which canalso be modi ed on-line by additional perceptual vari-ables. A simple extension allows to cope with mul-tiple degrees-of-freedom (DOF) patterns, where allDOFs share the same fundamental frequency but,otherwise, can move in arbitrary phase and ampli-tude o sets to each other. We evaluate our meth-ods in learning from demonstration with an actual30 DOF humanoid robot. Figure-8 and drummingmovements are demonstrated by a human, recordedin joint angle space with an exoskeleton, and em-bedded in multi-dimensional rhythmic pattern gener-ators. The learned patterns can be used by the robotin various workspace locations and from arbitraryinitial conditions. Spatial and temporal invarianceof the pattern generators allow easy amplitude andspeed scaling without losing the qualitative signatureof a movement. This novel way of creating rhyth-mic patterns could tremendously facilitate rhythmicmovement generation, in particular in locomotion ofrobots and neural prosthetics in clinical applications.
---
paper_title: Learning deep neural network policies with continuous memory states

paper_content:

Policy learning for partially observed control tasks requires policies that can remember salient information from past observations. In this paper, we present a method for learning policies with internal memory for high-dimensional, continuous systems, such as robotic manipulators. Our approach consists of augmenting the state and action space of the system with continuous-valued memory states that the policy can read from and write to. Learning general-purpose policies with this type of memory representation directly is difficult, because the policy must automatically figure out the most salient information to memorize at each time step. We show that, by decomposing this policy search problem into a trajectory optimization phase and a supervised learning phase through a method called guided policy search, we can acquire policies with effective memorization and recall strategies. Intuitively, the trajectory optimization phase chooses the values of the memory states that will make it easier for the policy to produce the right action in future states, while the supervised learning phase encourages the policy to use memorization actions to produce those memory states. We evaluate our method on tasks involving continuous control in manipulation and navigation settings, and show that our method can learn complex policies that successfully complete a range of tasks that require memory.
---
paper_title: Reinforcement learning of motor skills with policy gradients

paper_content:

Autonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm.
---
paper_title: Skill learning and task outcome prediction for manipulation

paper_content:

Learning complex motor skills for real world tasks is a hard problem in robotic manipulation that often requires painstaking manual tuning and design by a human expert. In this work, we present a Reinforcement Learning based approach to acquiring new motor skills from demonstration. Our approach allows the robot to learn fine manipulation skills and significantly improve its success rate and skill level starting from a possibly coarse demonstration. Our approach aims to incorporate task domain knowledge, where appropriate, by working in a space consistent with the constraints of a specific task. In addition, we also present an approach to using sensor feedback to learn a predictive model of the task outcome. This allows our system to learn the proprioceptive sensor feedback needed to monitor subsequent executions of the task online and abort execution in the event of predicted failure. We illustrate our approach using two example tasks executed with the PR2 dual-arm robot: a straight and accurate pool stroke and a box flipping task using two chopsticks as tools.
---
paper_title: Policy gradient reinforcement learning for fast quadrupedal locomotion

paper_content:

This paper presents a machine learning approach to optimizing a quadrupedal trot gait for forward speed. Given a parameterized walk designed for a specific robot, we propose using a form of policy gradient reinforcement learning to automatically search the set of possible parameters with the goal of finding the fastest possible walk. We implement and test our approach on a commercially available quadrupedal robot platform, namely the Sony Aibo robot. After about three hours of learning, all on the physical robots and with no human intervention other than to change the batteries, the robots achieved a gait faster than any previously known gait known for the Aibo, significantly outperforming a variety of existing hand-coded and learned solutions.
---
paper_title: Learning variable impedance control

paper_content:

One of the hallmarks of the performance, versatility, and robustness of biological motor control is the ability to adapt the impedance of the overall biomechanical system to different task requirements and stochastic disturbances. A transfer of this principle to robotics is desirable, for instance to enable robots to work robustly and safely in everyday human environments. It is, however, not trivial to derive variable impedance controllers for practical high degree-of-freedom (DOF) robotic tasks. In this contribution, we accomplish such variable impedance control with the reinforcement learning (RL) algorithm PI2 (Policy Improvement with Path Integrals). PI2 is a model-free, sampling-based learning method derived from first principles of stochastic optimal control. The PI 2 algorithm requires no tuning of algorithmic parameters besides the exploration noise. The designer can thus fully focus on the cost function design to specify the task. From the viewpoint of robotics, a particular useful property of PI2 is that it can scale to problems of many DOFs, so that reinforcement learning on real robotic systems becomes feasible. We sketch the PI2 algorithm and its theoretical properties, and how it is applied to gain scheduling for variable impedance control. We evaluate our approach by presenting results on several simulated and real robots. We consider tasks involving accurate tracking through via points, and manipulation tasks requiring physical contact with the environment. In these tasks, the optimal strategy requires both tuning of a reference trajectory and the impedance of the end-effector. The results show that we can use path integral based reinforcement learning not only for planning but also to derive variable gain feedback controllers in realistic scenarios. Thus, the power of variable impedance control is made available to a wide variety of robotic systems and practical applications.
---
paper_title: Reinforcement learning for imitating constrained reaching movements

paper_content:

The goal of developing algorithms for programming robots by demonstration is to create an easy way of programming robots that can be accomplished by everyone. When a demonstrator teaches a task to a robot, he/she shows some ways of fulfilling the task, but not all the possibilities. The robot must then be able to reproduce the task even when unexpected perturbations occur. In this case, it has to learn a new solution. In this paper, we describe a system to teach to the robot constrained reaching tasks. Our system is based on a dynamical system generator modulated with a learned speed trajectory and combined with a reinforcement learning module to allow the robot to adapt the trajectory when facing a new situation, such as avoiding obstacles.
---
paper_title: Guided Policy Search

paper_content:

Direct policy search can effectively scale to high-dimensional systems, but complex policies with hundreds of parameters often present a challenge for such methods, requiring numerous samples and often falling into poor local optima. We present a guided policy search algorithm that uses trajectory optimization to direct policy learning and avoid poor local optima. We show how differential dynamic programming can be used to generate suitable guiding samples, and describe a regularized importance sampled policy optimization that incorporates these samples into the policy search. We evaluate the method by learning neural network controllers for planar swimming, hopping, and walking, as well as simulated 3D humanoid running.
---
paper_title: Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning

paper_content:

The combination of modern Reinforcement Learning and Deep Learning approaches holds the promise of making significant progress on challenging applications requiring both rich perception and policy-selection. The Arcade Learning Environment (ALE) provides a set of Atari games that represent a useful benchmark set of such applications. A recent breakthrough in combining model-free reinforcement learning with deep learning, called DQN, achieves the best real-time agents thus far. Planning-based approaches achieve far higher scores than the best model-free approaches, but they exploit information that is not available to human players, and they are orders of magnitude slower than needed for real-time play. Our main goal in this work is to build a better real-time Atari game playing agent than DQN. The central idea is to use the slow planning-based agents to provide training data for a deep-learning architecture capable of real-time play. We proposed new agents based on this idea and show that they outperform DQN.
---
paper_title: Towards a Simulator for Imitation Learning with Kinesthetic Bootstrapping

paper_content:

This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented.
---
paper_title: Prototyping a biped robot using an educational robotics kit

paper_content:

This paper reports an experience of use of a robotics educational kit to prototype a biped robot, the robot was used to validate a new walking trajectory generation approach. The use of robotic kits is an alternative for low-cost prototyping and testing robotic solutions. Even if these robots did not meet the requirements of an industrial design, they can offer a first validation alternative. In this work, the Bioloid robotic kit is used to implement and evaluate a new algorithm for biped gait generation called IK-PSO. The algorithm used a forward kinematics model and a target position of the biped center of mass, COM, to generate an inverse kinematics solution of the joints motions. The remaining of the control process is a classical one. The pre-computed position of the Centre of mass, COM, allowed the robot to walk with a high degree of confidence that its stability is guaranteed. The educational robotics kit used allowed a fast prototyping and a practical test of the proposed algorithm.
---
paper_title: UAV path planning method based on ant colony optimization

paper_content:

A new UAV path planning method based on ant colony optimization (ACO) is presented. The target position is considered as the food source which the ants are going to find. The enemy defense region is considered as the searching area of the ants and is divided into equally spaced grids. The ants move to the destination node through several nodes on the grid region. The visibility function of ACO algorithm considers the enemy threats intensity on the paths and the distance to the destination node. The weighted sums of the flight path length, the threat cost and the maximum restriction of the yaw angle are considered as the evaluation function of ACO algorithm. The pheromone amounts on the paths are updated according to the evaluation function values. Therefore, the UAV optimal flight path is expressed by a group of node number, which is obtained by the ants finding the optimal route to the food source. The ACO algorithm based UAV path planning method is characterized as simple coding and good optimization guidance, and the simulation results also show its effectiveness.
---
paper_title: Towards automatic personalised content creation for racing games

paper_content:

Evolutionary algorithms are commonly used to create high-performing strategies or agents for computer games. In this paper, we instead choose to evolve the racing tracks in a car racing game. An evolvable track representation is devised, and a multiobjective evolutionary algorithm maximises the entertainment value of the track relative to a particular human player. This requires a way to create accurate models of players' driving styles, as well as a tentative definition of when a racing track is fun, both of which are provided. We believe this approach opens up interesting new research questions and is potentially applicable to commercial racing games.
---
paper_title: A Comprehensive Survey on Particle Swarm Optimization Algorithm and Its Applications

paper_content:

Particle swarm optimization (PSO) is a heuristic global optimization method, proposed originally by Kennedy and Eberhart in 1995. It is now one of the most commonly used optimization techniques. This survey presented a comprehensive investigation of PSO. On one hand, we provided advances with PSO, including its modifications (including quantum-behaved PSO, bare-bones PSO, chaotic PSO, and fuzzy PSO), population topology (as fully connected, von Neumann, ring, star, random, etc.), hybridization (with genetic algorithm, simulated annealing, Tabu search, artificial immune system, ant colony algorithm, artificial bee colony, differential evolution, harmonic search, and biogeography-based optimization), extensions (to multiobjective, constrained, discrete, and binary optimization), theoretical analysis (parameter selection and tuning, and convergence analysis), and parallel implementation (in multicore, multiprocessor, GPU, and cloud computing forms). On the other hand, we offered a survey on applications of PSO to the following eight fields: electrical and electronic engineering, automation control systems, communication theory, operations research, mechanical engineering, fuel and energy, medicine, chemistry, and biology. It is hoped that this survey would be beneficial for the researchers studying PSO algorithms.
---
paper_title: Obstacle avoidance with multi-objective optimization by PSO in dynamic environment

paper_content:

The second order motion model is one of the fundamental questions, a mostly important object in motion planning research of mobile robots, especially in complex environment. Based on the research of the second order motion model, this paper puts forward a new method for adjusting robots to avoid obstacles in dynamic environment. A mathematical model is first established in which environmental information such as, destination of a mobile robot, velocity and direction of obstacles are considered. Secondly, a new particle swarm optimization (PSO) algorithm is used to search for solution of the multi-objective optimization problem as described in the mathematical model. Finally, by adjusting the velocity and direction of the mobile robot to avoid obstacles in real time, the robot can reach the goal safely. Simulation experiment shows that this method is better than tradition artificial potential field (APF) algorithm and its improved algorithm based on genetic algorithm for obstacle avoidance.
---
paper_title: Imitating human playing styles in Super Mario Bros

paper_content:

Abstract We describe and compare several methods for generating game character controllers that mimic the playing style of a particular human player, or of a population of human players, across video game levels. Similarity in playing style is measured through an evaluation framework, that compares the play trace of one or several human players with the punctuated play trace of an AI player. The methods that are compared are either hand-coded, direct (based on supervised learning) or indirect (based on maximising a similarity measure). We find that a method based on neuroevolution performs best both in terms of the instrumental similarity measure and in phenomenological evaluation by human spectators. A version of the classic platform game “Super Mario Bros” is used as the testbed game in this study but the methods are applicable to other games that are based on character movement in space.
---
paper_title: A social learning particle swarm optimization algorithm for scalable optimization

paper_content:

Social learning plays an important role in behavior learning among social animals. In contrast to individual (asocial) learning, social learning has the advantage of allowing individuals to learn behaviors from others without incurring the costs of individual trials-and-errors. This paper introduces social learning mechanisms into particle swarm optimization (PSO) to develop a social learning PSO (SL-PSO). Unlike classical PSO variants where the particles are updated based on historical information, including the best solution found by the whole swarm (global best) and the best solution found by each particle (personal best), each particle in the proposed SL-PSO learns from any better particles (termed demonstrators) in the current swarm. In addition, to ease the burden of parameter settings, the proposed SL-PSO adopts a dimension-dependent parameter control method. The proposed SL-PSO is first compared with five representative PSO variants on 40 low-dimensional test functions, including shifted and rotated test functions. The scalability of the proposed SL-PSO is further tested by comparing it with five state-of-the-art algorithms for large-scale optimization on seven high-dimensional (100-D, 500-D, and 1000-D) benchmark functions. Our comparative results show that SL-PSO performs well on low-dimensional problems and is promising for solving large-scale problems as well.
---
paper_title: Optimal UAV flight path planning using skeletonization and Particle Swarm Optimizer

paper_content:

The purpose of this paper is to search the best flight route efficiently for unmanned aerial vehicle (UAV) in the 3-dimention complicated topography. The proposed method for the best flight route is mainly utilizing evolutionary algorithm, and give the proper initial population of evolutionary algorithm through skeletonization, efficient pre-processing procedure. In order to provide a smooth flight route for UAV, this paper adopts B-spline Curve method. Several control points of B-spline Curve method must be determined to generate flight route. The best control points can be calculated by Particle Swarm Optimizer (PSO). In this paper, the initial population of PSO is provided by skeletonization. The skeletonization of pre-processing procedure mainly includes two parts: one is Skeletonization and the other is candidate path searching. The purpose of pre-processing procedure is to reduce computation time, to prevent search the best solutions aimless, and execute evolutionary process efficiently. This paper uses Matlab as the experiment environment. The results of the experiments present the proposed method can provide the best flight route for UAV efficiently.
---
paper_title: Combining fitness-based search and user modeling in evolutionary robotics

paper_content:

Methodologies are emerging in many branches of computer science that demonstrate how human users and automated algorithms can collaborate on a problem such that their combined solutions outperform those produced by either humans or algorithms alone. The problem of behavior optimization in robotics seems particularly well-suited for this approach because humans have intuitions about how animals---and thus robots---should and should not behave, and can visually detect non-optimal behaviors that are trapped in local optima. Here we introduce a multiobjective approach in which a surrogate user (which stands in for a human user) deflects search away from local optima and a traditional fitness function eventually leads search toward the global optimum. We show that this approach produces superior solutions for a deceptive robotics problem compared to a similar search method that is guided by just a surrogate user or just a fitness function.
---
paper_title: Correcting and improving imitation models of humans for Robosoccer agents

paper_content:

The Robosoccer simulator is a challenging environment, where a human introduces a team of agents into a football virtual environment. Typically, agents are programmed by hand, but it would be a great advantage to transfer human experience into football agents. The first aim of this paper is to use machine learning techniques to obtain models of humans playing Robosoccer. These models can be used later to control a Robosoccer agent. However, models did not play as smoothly and optimally as the human. To solve this problem, the second goal of this paper is to incrementally correct models by means of evolutionary techniques, and to adapt them against more difficult opponents than the ones beatable by the human.
---
paper_title: Cellular Encoding for Interactive Evolutionary Robotics

paper_content:

This work reports experiments in interactive evolutionary robotics. The goal is to evolve an Artificial Neural Network (ANN) to control the locomotion of an 8-legged robot. The ANNs are encoded using a cellular developmental process called cellular encoding. In a previous work similar experiments have been carried on successfully on a simulated robot. They took however around 1,000,000 different ANN evaluations. In this work the fitness is determined on a real robot, and no more than a few hundreds evaluations can be performed. Various ideas were implemented so as to decrease the required number of evaluations from 1,000,000 to 200. First we used cell cloning and link typing. Second we did as many things as possible interactively: interactive problem decomposition, interactive syntactic constraints, interactive fitness. More precisely: 1- A modular design was chosen where a controller for an individual leg, with a precise neuronal interface was developed. 2- Syntactic constraints were used to promoting useful building blocs and impose an 8-fold symmetry. 3- We determine the fitness interactively by hand. We can reward features that would otherwise be very difficult to locate automatically. Interactive evolutionary robotics turns out to be quite successful, in the first bug-free run a global locomotion controller that is faster than a programmed controller could be evolved.
---
paper_title: Evaluation of human-robot arm movement imitation

paper_content:

Robot skill learning by imitation is an intuitive approach to learn robot skills from the observation of human behaviors. However, due to the discrepancy of mechanism between humans and robots such as the type of joint and the number of degree of freedom (DOF), a robot may not be able to imitate a human's movements faithfully. For a robot arm, the major problem is caused by the lack of a degree of freedom in the shoulder compared to a human arm. In this paper, we develop a similarity metric to evaluate how faithfully a robot arm imitates a human's arm movements. This metric is derived by utilizing the sequence-independent joint angle representation for both human and robot arms because it represents their postures more directly than the sequence-dependent Euler joint angle representation. In addition, the derived metric is formulated with the spatial relationship between human and robot arm postures instead of the Frobenius norm of the difference matrix between human and robot transformation matrices. To investigate the joint angles of the sequence-independent joint angle representation for a human arm, we adopt the particle-swarm optimization (PSO) to numerically derive them from human demonstration data. Computer simulations and experimental work were conducted to validate the proposed approach on a robot arm with two degrees of freedom in the shoulder and a DOF in the elbow.
---
paper_title: Graph-based domain mapping for transfer learning in general games

paper_content:

A general game player is an agent capable of taking as input a description of a game's rules in a formal language and proceeding to play without any subsequent human input. To do well, an agent should learn from experience with past games and transfer the learned knowledge to new problems. We introduce a graph-based method for identifying previously encountered games and prove its robustness formally. We then describe how the same basic approach can be used to identify similar but non-identical games. We apply this technique to automate domain mapping for value function transfer and speed up reinforcement learning on variants of previously played games. Our approach is fully implemented with empirical results in the general game playing system.
---
paper_title: Policy Transfer using Reward Shaping

paper_content:

Transfer learning has proven to be a wildly successful approach for speeding up reinforcement learning. Techniques often use low-level information obtained in the source task to achieve successful transfer in the target task. Yet, a most general transfer approach can only assume access to the output of the learning algorithm in the source task, i.e. the learned policy, enabling transfer irrespective of the learning algorithm used in the source task. We advance the state-of-the-art by using a reward shaping approach to policy transfer. One of the advantages in following such an approach, is that it firmly grounds policy transfer in an actively developing body of theoretical research on reward shaping. Experiments in Mountain Car, Cart Pole and Mario demonstrate the practical usefulness of the approach.
---
paper_title: Using advice to transfer knowledge acquired in one reinforcement learning task to another

paper_content:

We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice.
---
paper_title: Maximum entropy inverse reinforcement learning

paper_content:

Recent research has shown the benefit of framing problems of imitation learning as solutions to Markov Decision Problems. This approach reduces learning to the problem of recovering a utility function that makes the behavior induced by a near-optimal policy closely mimic demonstrated behavior. In this work, we develop a probabilistic approach based on the principle of maximum entropy. Our approach provides a well-defined, globally normalized distribution over decision sequences, while providing the same performance guarantees as existing methods. ::: ::: We develop our technique in the context of modeling real-world navigation and driving behaviors where collected data is inherently noisy and imperfect. Our probabilistic approach enables modeling of route preferences as well as a powerful new approach to inferring destinations and routes based on partial trajectories.
---
paper_title: Apprenticeship learning via inverse reinforcement learning

paper_content:

We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function.
---
paper_title: Maximum Entropy Deep Inverse Reinforcement Learning

paper_content:

This paper presents a general framework for exploiting the representational capacity of neural networks to approximate complex, nonlinear reward functions in the context of solving the inverse reinforcement learning (IRL) problem. We show in this context that the Maximum Entropy paradigm for IRL lends itself naturally to the efficient training of deep architectures. At test time, the approach leads to a computational complexity independent of the number of demonstrations, which makes it especially well-suited for applications in life-long learning scenarios. Our approach achieves performance commensurate to the state-of-the-art on existing benchmarks while exceeding on an alternative benchmark based on highly varying reward structures. Finally, we extend the basic architecture - which is equivalent to a simplified subclass of Fully Convolutional Neural Networks (FCNNs) with width one - to include larger convolutions in order to eliminate dependency on precomputed spatial features and work on raw input representations.
---
paper_title: Teaching collaborative multi-robot tasks through demonstration

paper_content:

Humanoid robots working alongside humans in everyday environments is a long standing goal of the robotics community. To achieve this goal, methods for developing new robot behaviors that are intuitive and accessible to non-programmers are required. In this paper, we present a demonstration-based method for teaching distributed autonomous robots to coordinate their actions and perform collaborative multi-robot tasks. Within the presented framework, each robot learns an individual policy from teacher demonstrations using a confidence-based algorithm. Based on this learning approach, we contribute three techniques for teaching multi-robot coordination using different information sharing strategies. We evaluate and compare these approaches by teaching two Sony QRIO humanoid robots to perform three collaborative ball sorting tasks.
---
paper_title: What is the Teacher’s Role in Robot Programming by Demonstration? Toward Benchmarks for Improved Learning

paper_content:

Robot programming by demonstration (RPD) covers methods by which a robot learns new skills through human guidance. We present an interactive, multimodal RPD framework using active teaching methods that places the human teacher in the robot's learning loop. Two experiments are presented in which observational learning is first used to demonstrate a manipulation skill to a HOAP-3 humanoid robot by using motion sensors attached to the teacher's body. Then, putting the robot through the motion, the teacher incrementally refines the robot's skill by moving its arms manually, providing the appropriate scaffolds to reproduce the action. An incremental teaching scenario is proposed based on insights from various fields addressing developmental, psychological, and social issues related to teaching mechanisms in humans. Based on this analysis, different benchmarks are suggested to evaluate the setup further.
---
paper_title: Confidence-based policy learning from demonstration using Gaussian mixture models

paper_content:

We contribute an approach for interactive policy learning through expert demonstration that allows an agent to actively request and effectively represent demonstration examples. In order to address the inherent uncertainty of human demonstration, we represent the policy as a set of Gaussian mixture models (GMMs), where each model, with multiple Gaussian components, corresponds to a single action. Incrementally received demonstration examples are used as training data for the GMM set. We then introduce our confident execution approach, which focuses learning on relevant parts of the domain by enabling the agent to identify the need for and request demonstrations for specific parts of the state space. The agent selects between demonstration and autonomous execution based on statistical analysis of the uncertainty of the learned Gaussian mixture set. As it achieves proficiency at its task and gains confidence in its actions, the agent operates with increasing autonomy, eliminating the need for unnecessary demonstrations of already acquired behavior, and reducing both the training time and the demonstration workload of the expert. We validate our approach with experiments in simulated and real robot domains.
---
paper_title: Physical Human-Robot Interaction: Mutual Learning and Adaptation

paper_content:

Close physical interaction between robots and humans is a particularly challenging aspect of robot development. For successful interaction and cooperation, the robot must have the ability to adapt its behavior to the human counterpart. Based on our earlier work, we present and evaluate a computationally efficient machine learning algorithm that is well suited for such close-contact interaction scenarios. We show that this algorithm helps to improve the quality of the interaction between a robot and a human caregiver. To this end, we present two human-in-the-loop learning scenarios that are inspired by human parenting behavior, namely, an assisted standing-up task and an assisted walking task.
---
paper_title: Smooth Imitation Learning for Online Sequence Prediction

paper_content:

We study the problem of smooth imitation learning for online sequence prediction, where the goal is to train a policy that can smoothly imitate demonstrated behavior in a dynamic and continuous environment in response to online, sequential context input. Since the mapping from context to behavior is often complex, we take a learning reduction approach to reduce smooth imitation learning to a regression problem using complex function classes that are regularized to ensure smoothness. We present a learning meta-algorithm that achieves fast and stable convergence to a good policy. Our approach enjoys several attractive properties, including being fully deterministic, employing an adaptive learning rate that can provably yield larger policy improvements compared to previous approaches, and the ability to ensure stable convergence. Our empirical results demonstrate significant performance gains over previous approaches.
---
paper_title: Search-based structured prediction

paper_content:

We present Searn, an algorithm for integrating search and learning to solve complex structured prediction problems such as those that occur in natural language, speech, computational biology, and vision. Searn is a meta-algorithm that transforms these complex problems into simple classification problems to which any binary classifier may be applied. Unlike current algorithms for structured learning that require decomposition of both the loss function and the feature functions over the predicted structure, Searn is able to learn prediction functions for any loss function and any class of features. Moreover, Searn comes with a strong, natural theoretical guarantee: good performance on the derived classification problems implies good performance on the structured prediction problem.
---
paper_title: Learning a repertoire of actions with deep neural networks

paper_content:

We address the problem of endowing a robot with the capability to learn a repertoire of actions using as little prior knowledge as possible. Taking a handwriting task as an example, we apply the deep learning paradigm to build a network which uses a high-level representation of digits to generate sequences of commands, directly fed to a low-level control loop. Discrete variables are used to discriminate different digits, while continuous variables parametrize each digit. We show that the proposed network is able to generalize learned actions to new contexts. The network is tested on trajectories recorded on the iCub humanoid robot.
---
paper_title: A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural Networks

paper_content:

Tying suture knots is a time-consuming task performed frequently during minimally invasive surgery (MIS). Automating this task could greatly reduce total surgery time for patients. Current solutions to this problem replay manually programmed trajectories, but a more general and robust approach is to use supervised machine learning to smooth surgeon-given training trajectories and generalize from them. Since knottying generally requires a controller with internal memory to distinguish between identical inputs that require different actions at different points along a trajectory, it would be impossible to teach the system using traditional feedforward neural nets or support vector machines. Instead we exploit more powerful, recurrent neural networks (RNNs) with adaptive internal states. Results obtained using LSTM RNNs trained by the recent Evolino algorithm show that this approach can significantly increase the efficiency of suture knot tying in MIS over preprogrammed control
---
paper_title: Teaching collaborative multi-robot tasks through demonstration

paper_content:

Humanoid robots working alongside humans in everyday environments is a long standing goal of the robotics community. To achieve this goal, methods for developing new robot behaviors that are intuitive and accessible to non-programmers are required. In this paper, we present a demonstration-based method for teaching distributed autonomous robots to coordinate their actions and perform collaborative multi-robot tasks. Within the presented framework, each robot learns an individual policy from teacher demonstrations using a confidence-based algorithm. Based on this learning approach, we contribute three techniques for teaching multi-robot coordination using different information sharing strategies. We evaluate and compare these approaches by teaching two Sony QRIO humanoid robots to perform three collaborative ball sorting tasks.
---
paper_title: Implicit Imitation in Multiagent Reinforcement Learning

paper_content:

Imitation is actively being studied as an effective means of learning in multi-agent environments. It allows an agent to learn how to act well (perhaps optimally) by passively observing the actions of cooperative teachers or other more experienced agents its environment. We propose a straightforward imitation mechanism called model extraction that can be integrated easily into standard model-based reinforcement learning algorithms. Roughly, by observing a mentor with similar capabilities, an agent can extract information about its own capabilities in unvisited parts of state space. The extracted information can accelerate learning dramatically. We illustrate the benefits of model extraction by integrating it with prioritized sweeping, and demonstrating improved performance and convergence through observation of single and multiple mentors. Though we make some stringent assumptions regarding observability, possible interactions and common abilities, we briefly comment on extensions of the model that relax these.
---
paper_title: Multi-Agent Reinforcement Learning: a critical survey

paper_content:

We survey the recent work in AI on multi-agent reinforcement learning (that is, learning in stochastic games). We then argue that, while exciting, this work is flawed. The fundamental flaw is unclarity about the problem or problems being addressed. After tracing a representative sample of the recent literature, we identify four well-defined problems in multi-agent reinforcement learning, single out the problem that in our view is most suitable for AI, and make some remarks about how we believe progress is to be made on this problem.
---
paper_title: Teaching coordinated strategies to soccer robots via imitation

paper_content:

Developing coordination among multiple agents and enabling them to exhibit teamwork is a challenging yet exciting task that can benefit many of the complex real-life problems. This research uses imitation to learn collaborative strategies for a team of agents. Imitation based learning involves learning from an expert by observing him/her demonstrating a task and then replicating it. The key idea is to involve multiple human experts during demonstration to teach autonomous agents how to work in coordination. The effectiveness of the proposed methodology has been assessed in a goal defending scenario of the RoboCup Soccer Simulation 3D league. The process involves multiple human demonstrators controlling soccer agents via game controllers and demonstrating them how to play soccer in coordination. The data gathered during this phase is used as training data to learn a classification model which is later used by the soccer agents to make autonomous decisions during actual matches. Different performance evaluation metrics are derived to compare the performance of imitating agent with that of the human-driven agent and hand-coded (if-then-else rules) agent.
---
paper_title: Imitation learning at all levels of game-AI

paper_content:

Imitation is a powerful mechanism the human brain applies to extend its repertoire of solutions and behaviors suitable to solve problems of various kinds. From an abstract point of view, the major advantage of this strategy is that it reduces the search space of apropriate solutions. In this contribution, we discuss if and how the principle of imitation learning can facilitate the programing of life-like computer game charecters. We present different algorithms that learn from human generated training data and we show that machine learning can be applied on different levels of cognitive abstraction.
---
paper_title: Imitation learning through games: theory, implementation and evaluation

paper_content:

Despite a history of games-based research, academia has generally regarded ::: commercial games as a distraction from the serious business of AI, rather than as an ::: opportunity to leverage this existing domain to the advancement of our knowledge. ::: Similarly, the computer game industry still relies on techniques that were developed ::: several decades ago, and has shown little interest in adopting more progressive ::: academic approaches. In recent times, however, these attitudes have begun to change; ::: under- and post-graduate games development courses are increasingly common, ::: while the industry itself is slowly but surely beginning to recognise the potential ::: offered by modern machine-learning approaches, though games which actually ::: implement said approaches on more than a token scale remain scarce. ::: One area which has not yet received much attention from either academia or industry ::: is imitation learning, which seeks to expedite the learning process by exploiting data ::: harvested from demonstrations of a given task. While substantial work has been done ::: in developing imitation techniques for humanoid robot movement, there has been ::: very little exploration of the challenges posed by interactive computer games. Given ::: that such games generally encode reasoning and decision-making behaviours which ::: are inherently more complex and potentially more interesting than limb motion data, ::: that they often provide inbuilt facilities for recording human play, that the generation ::: and collection of training samples is therefore far easier than in robotics, and that ::: many games have vast pre-existing libraries of these recorded demonstrations, it is ::: fair to say that computer games represent an extremely fertile domain for imitation ::: learning research. ::: In this thesis, we argue in favour of using modern, commercial computer games to ::: study, model and reproduce humanlike behaviour. We provide an overview of the ::: biological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based ::: imitation. We then proceed to describe our contributions to the field of imitation ::: learning itself, which encompass three distinct categories: theory, implementation ::: and evaluation. ::: We first describe the development of a fully-featured Java API - the Quake2 Agent ::: Simulation Environment (QASE) - designed to facilitate both research and education ::: in imitation and general machine-learning, using the game Quake 2 as a testbed. We ::: outline our motivation for developing QASE, discussing the shortcomings of existing ::: APIs and the steps which we have taken to circumvent them. We describe QASE’s ::: network layer, which acts as an interface between the local AI routines and the ::: Quake 2 server on which the game environment is maintained, before detailing the ::: API’s agent architecture, which includes an interface to the MatLab programming ::: environment and the ability to parse and analyse full recordings of game sessions. ::: We conclude the chapter with a discussion of QASE’s adoption by numerous ::: universities as both an undergraduate teaching tool and research platform. ::: We then proceed to describe the various imitative mechanisms which we have ::: developed using QASE and its MatLab integration facilities. We first outline a ::: behaviour model based on a well-known psychological model of human planning. ::: Drawing upon previous research, we also identify a set of believability criteria - ::: elements of agent behaviour which are of particular importance in determining the ::: “humanness” of its in-game appearance. We then detail a reinforcement-learning ::: approach to imitating the human player’s navigation of his environment, centred ::: upon his pursuit of items as strategic goals. In the subsequent section, we describe ::: the integration of this strategic system with a Bayesian mechanism for the imitation ::: of tactical and motion-modelling behaviours. Finally, we outline a model for the ::: imitation of reactive combat behaviours; specifically, weapon-selection and aiming. Experiments are presented in each case to demonstrate the imitative mechanisms’ ::: ability to accurately reproduce observed behaviours. ::: Finally, we criticise the lack of any existing methodology to formally gauge the ::: believability of game agents, and observe that the few previous attempts have been ::: extremely ad-hoc and informal. We therefore propose a generalised approach to such ::: testing; the Bot-Oriented Turing Test (BOTT). This takes the form of an anonymous ::: online questionnaire, an accompanying protocol to which examiners should adhere, ::: and the formulation of a believability index which numerically expresses each agent’s ::: humanness as indicated by its observers, weighted by their experience and the ::: accuracy with which the agents were identified. To both validate the survey approach ::: and to determine the efficacy of our imitative models, we present a series of ::: experiments which use the believability test to evaluate our own imitation agents ::: against both human players and traditional artificial bots. We demonstrate that our ::: imitation agents perform substantially better than even a highly-regarded rule-based ::: agent, and indeed approach the believability of actual human players. ::: Some suggestions for future directions in our research, as well as a broader ::: discussion of open questions, conclude this thesis.
---
paper_title: Teaching coordinated strategies to soccer robots via imitation

paper_content:

Developing coordination among multiple agents and enabling them to exhibit teamwork is a challenging yet exciting task that can benefit many of the complex real-life problems. This research uses imitation to learn collaborative strategies for a team of agents. Imitation based learning involves learning from an expert by observing him/her demonstrating a task and then replicating it. The key idea is to involve multiple human experts during demonstration to teach autonomous agents how to work in coordination. The effectiveness of the proposed methodology has been assessed in a goal defending scenario of the RoboCup Soccer Simulation 3D league. The process involves multiple human demonstrators controlling soccer agents via game controllers and demonstrating them how to play soccer in coordination. The data gathered during this phase is used as training data to learn a classification model which is later used by the soccer agents to make autonomous decisions during actual matches. Different performance evaluation metrics are derived to compare the performance of imitating agent with that of the human-driven agent and hand-coded (if-then-else rules) agent.
---
paper_title: Human-level control through deep reinforcement learning

paper_content:

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
---
paper_title: Imitation learning by coaching

paper_content:

Imitation Learning has been shown to be successful in solving many challenging real-world problems. Some recent approaches give strong performance guarantees by training the policy iteratively. However, it is important to note that these guarantees depend on how well the policy we found can imitate the oracle on the training data. When there is a substantial difference between the oracle's ability and the learner's policy space, we may fail to find a policy that has low error on the training set. In such cases, we propose to use a coach that demonstrates easy-to-learn actions for the learner and gradually approaches the oracle. By a reduction of learning by demonstration to online learning, we prove that coaching can yield a lower regret bound than using the oracle. We apply our algorithm to cost-sensitive dynamic feature selection, a hard decision problem that considers a user-specified accuracy-cost trade-off. Experimental results on UCI datasets show that our method outperforms state-of-the-art imitation learning methods in dynamic feature selection and two static feature selection methods.
---
paper_title: Correcting and improving imitation models of humans for Robosoccer agents

paper_content:

The Robosoccer simulator is a challenging environment, where a human introduces a team of agents into a football virtual environment. Typically, agents are programmed by hand, but it would be a great advantage to transfer human experience into football agents. The first aim of this paper is to use machine learning techniques to obtain models of humans playing Robosoccer. These models can be used later to control a Robosoccer agent. However, models did not play as smoothly and optimally as the human. To solve this problem, the second goal of this paper is to incrementally correct models by means of evolutionary techniques, and to adapt them against more difficult opponents than the ones beatable by the human.
---
paper_title: Policy Search for Motor Primitives in Robotics

paper_content:

Many motor skills in humanoid robotics can be learned using parametrized motor primitives as done in imitation learning. However, most interesting motor learning problems are high-dimensional reinforcement learning problems often beyond the reach of current methods. In this paper, we extend previous work on policy learning from the immediate reward case to episodic reinforcement learning. We show that this results in a general, common framework also connected to policy gradient methods and yielding a novel algorithm for policy learning that is particularly well-suited for dynamic motor primitives. The resulting algorithm is an EM-inspired algorithm applicable to complex motor learning tasks. We compare this algorithm to several well-known parametrized policy search methods and show that it outperforms them. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task using a real Barrett WAM™ robot arm.
---
paper_title: Apprenticeship learning via inverse reinforcement learning

paper_content:

We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function.
---
paper_title: Computational Approaches to Motor Learning by Imitation

paper_content:

Movement imitation requires a complex set of mechanisms that map an observed movement of a teacher onto one's own movement apparatus. Relevant problems include movement recognition, pose estimation, pose tracking, body correspondence, coordinate transformation from external to egocentric space, matching of observed against previously learned movement, resolution of redundant degrees-of-freedom that are unconstrained by the observation, suitable movement representations for imitation, modularization of motor control, etc. All of these topics by themselves are active research problems in computational and neurobiological sciences, such that their combination into a complete imitation system remains a daunting undertaking-indeed, one could argue that we need to understand the complete perception-action loop. As a strategy to untangle the complexity of imitation, this paper will examine imitation purely from a computational point of view, i.e. we will review statistical and mathematical approaches that have been suggested for tackling parts of the imitation problem, and discuss their merits, disadvantages and underlying principles. Given the focus on action recognition of other contributions in this special issue, this paper will primarily emphasize the motor side of imitation, assuming that a perceptual system has already identified important features of a demonstrated movement and created their corresponding spatial information. Based on the formalization of motor control in terms of control policies and their associated performance criteria, useful taxonomies of imitation learning can be generated that clarify different approaches and future research directions.
---
paper_title: Using advice to transfer knowledge acquired in one reinforcement learning task to another

paper_content:

We present a method for transferring knowledge learned in one task to a related task. Our problem solvers employ reinforcement learning to acquire a model for one task. We then transform that learned model into advice for a new task. A human teacher provides a mapping from the old task to the new task to guide this knowledge transfer. Advice is incorporated into our problem solver using a knowledge-based support vector regression method that we previously developed. This advice-taking approach allows the problem solver to refine or even discard the transferred knowledge based on its subsequent experiences. We empirically demonstrate the effectiveness of our approach with two games from the RoboCup soccer simulator: KeepAway and BreakAway. Our results demonstrate that a problem solver learning to play BreakAway using advice extracted from KeepAway outperforms a problem solver learning without the benefit of such advice.
---
paper_title: Graph-based domain mapping for transfer learning in general games

paper_content:

A general game player is an agent capable of taking as input a description of a game's rules in a formal language and proceeding to play without any subsequent human input. To do well, an agent should learn from experience with past games and transfer the learned knowledge to new problems. We introduce a graph-based method for identifying previously encountered games and prove its robustness formally. We then describe how the same basic approach can be used to identify similar but non-identical games. We apply this technique to automate domain mapping for value function transfer and speed up reinforcement learning on variants of previously played games. Our approach is fully implemented with empirical results in the general game playing system.
---
paper_title: Imitating human playing styles in Super Mario Bros

paper_content:

Abstract We describe and compare several methods for generating game character controllers that mimic the playing style of a particular human player, or of a population of human players, across video game levels. Similarity in playing style is measured through an evaluation framework, that compares the play trace of one or several human players with the punctuated play trace of an AI player. The methods that are compared are either hand-coded, direct (based on supervised learning) or indirect (based on maximising a similarity measure). We find that a method based on neuroevolution performs best both in terms of the instrumental similarity measure and in phenomenological evaluation by human spectators. A version of the classic platform game “Super Mario Bros” is used as the testbed game in this study but the methods are applicable to other games that are based on character movement in space.
---
paper_title: Imitation learning through games: theory, implementation and evaluation

paper_content:

Despite a history of games-based research, academia has generally regarded ::: commercial games as a distraction from the serious business of AI, rather than as an ::: opportunity to leverage this existing domain to the advancement of our knowledge. ::: Similarly, the computer game industry still relies on techniques that were developed ::: several decades ago, and has shown little interest in adopting more progressive ::: academic approaches. In recent times, however, these attitudes have begun to change; ::: under- and post-graduate games development courses are increasingly common, ::: while the industry itself is slowly but surely beginning to recognise the potential ::: offered by modern machine-learning approaches, though games which actually ::: implement said approaches on more than a token scale remain scarce. ::: One area which has not yet received much attention from either academia or industry ::: is imitation learning, which seeks to expedite the learning process by exploiting data ::: harvested from demonstrations of a given task. While substantial work has been done ::: in developing imitation techniques for humanoid robot movement, there has been ::: very little exploration of the challenges posed by interactive computer games. Given ::: that such games generally encode reasoning and decision-making behaviours which ::: are inherently more complex and potentially more interesting than limb motion data, ::: that they often provide inbuilt facilities for recording human play, that the generation ::: and collection of training samples is therefore far easier than in robotics, and that ::: many games have vast pre-existing libraries of these recorded demonstrations, it is ::: fair to say that computer games represent an extremely fertile domain for imitation ::: learning research. ::: In this thesis, we argue in favour of using modern, commercial computer games to ::: study, model and reproduce humanlike behaviour. We provide an overview of the ::: biological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based ::: imitation. We then proceed to describe our contributions to the field of imitation ::: learning itself, which encompass three distinct categories: theory, implementation ::: and evaluation. ::: We first describe the development of a fully-featured Java API - the Quake2 Agent ::: Simulation Environment (QASE) - designed to facilitate both research and education ::: in imitation and general machine-learning, using the game Quake 2 as a testbed. We ::: outline our motivation for developing QASE, discussing the shortcomings of existing ::: APIs and the steps which we have taken to circumvent them. We describe QASE’s ::: network layer, which acts as an interface between the local AI routines and the ::: Quake 2 server on which the game environment is maintained, before detailing the ::: API’s agent architecture, which includes an interface to the MatLab programming ::: environment and the ability to parse and analyse full recordings of game sessions. ::: We conclude the chapter with a discussion of QASE’s adoption by numerous ::: universities as both an undergraduate teaching tool and research platform. ::: We then proceed to describe the various imitative mechanisms which we have ::: developed using QASE and its MatLab integration facilities. We first outline a ::: behaviour model based on a well-known psychological model of human planning. ::: Drawing upon previous research, we also identify a set of believability criteria - ::: elements of agent behaviour which are of particular importance in determining the ::: “humanness” of its in-game appearance. We then detail a reinforcement-learning ::: approach to imitating the human player’s navigation of his environment, centred ::: upon his pursuit of items as strategic goals. In the subsequent section, we describe ::: the integration of this strategic system with a Bayesian mechanism for the imitation ::: of tactical and motion-modelling behaviours. Finally, we outline a model for the ::: imitation of reactive combat behaviours; specifically, weapon-selection and aiming. Experiments are presented in each case to demonstrate the imitative mechanisms’ ::: ability to accurately reproduce observed behaviours. ::: Finally, we criticise the lack of any existing methodology to formally gauge the ::: believability of game agents, and observe that the few previous attempts have been ::: extremely ad-hoc and informal. We therefore propose a generalised approach to such ::: testing; the Bot-Oriented Turing Test (BOTT). This takes the form of an anonymous ::: online questionnaire, an accompanying protocol to which examiners should adhere, ::: and the formulation of a believability index which numerically expresses each agent’s ::: humanness as indicated by its observers, weighted by their experience and the ::: accuracy with which the agents were identified. To both validate the survey approach ::: and to determine the efficacy of our imitative models, we present a series of ::: experiments which use the believability test to evaluate our own imitation agents ::: against both human players and traditional artificial bots. We demonstrate that our ::: imitation agents perform substantially better than even a highly-regarded rule-based ::: agent, and indeed approach the believability of actual human players. ::: Some suggestions for future directions in our research, as well as a broader ::: discussion of open questions, conclude this thesis.
---
paper_title: What is the Teacher’s Role in Robot Programming by Demonstration? Toward Benchmarks for Improved Learning

paper_content:

Robot programming by demonstration (RPD) covers methods by which a robot learns new skills through human guidance. We present an interactive, multimodal RPD framework using active teaching methods that places the human teacher in the robot's learning loop. Two experiments are presented in which observational learning is first used to demonstrate a manipulation skill to a HOAP-3 humanoid robot by using motion sensors attached to the teacher's body. Then, putting the robot through the motion, the teacher incrementally refines the robot's skill by moving its arms manually, providing the appropriate scaffolds to reproduce the action. An incremental teaching scenario is proposed based on insights from various fields addressing developmental, psychological, and social issues related to teaching mechanisms in humans. Based on this analysis, different benchmarks are suggested to evaluate the setup further.
---
paper_title: What is the Teacher’s Role in Robot Programming by Demonstration? Toward Benchmarks for Improved Learning

paper_content:

Robot programming by demonstration (RPD) covers methods by which a robot learns new skills through human guidance. We present an interactive, multimodal RPD framework using active teaching methods that places the human teacher in the robot's learning loop. Two experiments are presented in which observational learning is first used to demonstrate a manipulation skill to a HOAP-3 humanoid robot by using motion sensors attached to the teacher's body. Then, putting the robot through the motion, the teacher incrementally refines the robot's skill by moving its arms manually, providing the appropriate scaffolds to reproduce the action. An incremental teaching scenario is proposed based on insights from various fields addressing developmental, psychological, and social issues related to teaching mechanisms in humans. Based on this analysis, different benchmarks are suggested to evaluate the setup further.
---
paper_title: Towards a Simulator for Imitation Learning with Kinesthetic Bootstrapping

paper_content:

This paper presents a physics based simulator that allows kinesthetic interactions between a human and a robot to be recorded, and later used for imitation learning. We argue that kinesthetic interac- tion can be a very important tool for programming robots, if properly supported by a simulation engine. For this, we propose a new scheme for robot motion learning based on kinesthetic bootstrapping. Interactions with the robot are used to create a low-dimensional posture space. The posture space together with the presented simulation engine allow for fast imitation learning of behaviors. Early results of this approach, using Genetic Algorithms as learning technique, will be presented.
---
paper_title: Learning drivers for TORCS through imitation using supervised methods

paper_content:

In this paper, we apply imitation learning to develop drivers for The Open Racing Car Simulator (TORCS). Our approach can be classified as a direct method in that it applies supervised learning to learn car racing behaviors from the data collected from other drivers. In the literature, this approach is known to have led to extremely poor performance with drivers capable of completing only very small parts of a track. In this paper we show that, by using high-level information about the track ahead of the car and by predicting high-level actions, it is possible to develop drivers with performances that in some cases are only 15% lower than the performance of the fastest driver available in TORCS. Our experimental results suggest that our approach can be effective in developing drivers with good performance in non-trivial tracks using a very limited amount of data and computational resources. We analyze the driving behavior of the controllers developed using our approach and identify perceptual aliasing as one of the factors which can limit performance of our approach.
---
paper_title: Recognizing teleoperated manipulations

paper_content:

The many degrees-of-freedom and distributed sensing capability of dextrous robot hands permit the use of control programs that rely on qualitative changes in sensor feedback. One way of designing such a control program is to have the robot learn the qualitative control characteristics from examples. These examples may be provided via teleoperation. Results are presented for recognizing and segmenting manipulation primitives from a teleoperated task by analysis of features in sensor feedback. k-nearest quantized pattern vectors determine potential classifications. A hidden Markov model provides task context for the final segmentation. The illustrative task is picking up a plastic egg with a spatula. >
---
paper_title: Confidence-based policy learning from demonstration using Gaussian mixture models

paper_content:

We contribute an approach for interactive policy learning through expert demonstration that allows an agent to actively request and effectively represent demonstration examples. In order to address the inherent uncertainty of human demonstration, we represent the policy as a set of Gaussian mixture models (GMMs), where each model, with multiple Gaussian components, corresponds to a single action. Incrementally received demonstration examples are used as training data for the GMM set. We then introduce our confident execution approach, which focuses learning on relevant parts of the domain by enabling the agent to identify the need for and request demonstrations for specific parts of the state space. The agent selects between demonstration and autonomous execution based on statistical analysis of the uncertainty of the learned Gaussian mixture set. As it achieves proficiency at its task and gains confidence in its actions, the agent operates with increasing autonomy, eliminating the need for unnecessary demonstrations of already acquired behavior, and reducing both the training time and the demonstration workload of the expert. We validate our approach with experiments in simulated and real robot domains.
---
paper_title: The use of socially assistive robots in the design of intelligent cognitive therapies for people with dementia

paper_content:

Currently the 2 percent growth rate for the world's older population exceeds the 1.2 rate for the world's population as a whole. By 2050, the number of individuals over the age 85 is projected to be three times more than there is today. Most of these individuals will need physical, emotional, and cognitive assistance. In this paper, we present a new adaptive robotic system based on the socially assistive robotics (SAR) technology that tries to provide a customized help protocol through motivation, encouragements, and companionship to users suffering from cognitive changes related to aging and/or Alzheimer's disease. Our results show that this approach can engage the patients and keep them interested in interacting with the robot, which, in turn, increases their positive behavior.
---
paper_title: Natural methods for robot task learning: instructive demonstrations, generalization and practice

paper_content:

Among humans, teaching various tasks is a complex process which relies on multiple means for interaction and learning, both on the part of the teacher and of the learner. Used together, these modalities lead to effective teaching and learning approaches, respectively. In the robotics domain, task teaching has been mostly addressed by using only one or very few of these interactions. In this paper we present an approach for teaching robots that relies on the key features and the general approach people use when teaching each other: first give a demonstration, then allow the learner to refine the acquired capabilities by practicing under the teacher's supervision, involving a small number of trials. Depending on the quality of the learned task, the teacher may either demonstrate it again or provide specific feedback during the learner's practice trial for further refinement. Also, as people do during demonstrations, the teacher can provide simple instructions and informative cues, increasing the performance of learning. Thus, instructive demonstrations, generalization over multiple demonstrations and practice trials are essential features for a successful human-robot teaching approach. We implemented a system that enables all these capabilities and validated these concepts with a Pioneer 2DX mobile robot learning tasks from multiple demonstrations and teacher feedback.
---
paper_title: Neural Network Vision for Robot Driving

paper_content:

Autonomous navigation is a difficult problem for traditional vision and robotic techniques, primarily because of the noise and variability associated with real world scenes. Autonomous navigation systems based on traditional image processing and pattern recognition techniques often perform well under certain conditions, but have problems with others. Part of the difficulty stems from the fact that the processing performed by these systems remains fixed across various environments.
---
paper_title: Imitation learning at all levels of game-AI

paper_content:

Imitation is a powerful mechanism the human brain applies to extend its repertoire of solutions and behaviors suitable to solve problems of various kinds. From an abstract point of view, the major advantage of this strategy is that it reduces the search space of apropriate solutions. In this contribution, we discuss if and how the principle of imitation learning can facilitate the programing of life-like computer game charecters. We present different algorithms that learn from human generated training data and we show that machine learning can be applied on different levels of cognitive abstraction.
---
paper_title: Believable Bots: Can Computers Play Like People?

paper_content:

We share our modern world with bots chatbots to converse with, roombots to clean our houses, spambots to fill our e-mail inboxes, and medibots to assist our surgeons. This book is about computer game bots, virtual companions who accompany us in virtual worlds or sharpen our fighting skills. These bots must be believable, that is human players should believe they are interacting with entities operating at a human level bots are more fun if they behave like we do. This book shows how to create believable bots that play computer games, and it discusses the implications of making them appear human. The chapters in this book present the state of the art in research on and development of game bots, and they also look beyond the design aspects to address deep questions: Is a bot that plays like a person intelligent? Does it have emotions? Is it conscious? The topic is inherently interdisciplinary, and the work draws from research and practice in many fields, such as design, creativity, entertainment, and graphics; learning, psychology, and sociology; artificial intelligence, embodiment, agents, machine learning, robotics, humancomputer interaction, and artificial life; cognition and neuroscience; and evolutionary computing. The contributing authors are among the leading researchers and developers in this field, and most of the examples and case studies involve analysis of commercial products. The book will be of value to graduate students and academic researchers in artificial intelligence, and to engineers charged with the design of entertaining games.
---
paper_title: Maximum entropy inverse reinforcement learning

paper_content:

Recent research has shown the benefit of framing problems of imitation learning as solutions to Markov Decision Problems. This approach reduces learning to the problem of recovering a utility function that makes the behavior induced by a near-optimal policy closely mimic demonstrated behavior. In this work, we develop a probabilistic approach based on the principle of maximum entropy. Our approach provides a well-defined, globally normalized distribution over decision sequences, while providing the same performance guarantees as existing methods. ::: ::: We develop our technique in the context of modeling real-world navigation and driving behaviors where collected data is inherently noisy and imperfect. Our probabilistic approach enables modeling of route preferences as well as a powerful new approach to inferring destinations and routes based on partial trajectories.
---
paper_title: Apprenticeship learning via inverse reinforcement learning

paper_content:

We consider learning in a Markov decision process where we are not explicitly given a reward function, but where instead we can observe an expert demonstrating the task that we want to learn to perform. This setting is useful in applications (such as the task of driving) where it may be difficult to write down an explicit reward function specifying exactly how different desiderata should be traded off. We think of the expert as trying to maximize a reward function that is expressible as a linear combination of known features, and give an algorithm for learning the task demonstrated by the expert. Our algorithm is based on using "inverse reinforcement learning" to try to recover the unknown reward function. We show that our algorithm terminates in a small number of iterations, and that even though we may never recover the expert's reward function, the policy output by the algorithm will attain performance close to that of the expert, where here performance is measured with respect to the expert's unknown reward function.
---
paper_title: Learning rhythmic movements by demonstration using nonlinear oscillators

paper_content:

ATR Human Information Science Laboratories, Kyoto, JapanEmail: ijspeert@usc.edu, jun@his.atr.co.jp, sschaal@usc.eduAbstractThis paper presents a new approach to the generationof rhythmic movement patterns with nonlinear dy-namical systems. Starting from a canonical limit cy-cle oscillator with well-de ned stability properties, wemodify the attractor landscape of the canonical sys-temby meansof statisticallearning methods toembedarbitrary smooth target patterns, however, withoutlosing the stability properties of the canonical sys-tem. In contrast to non-autonomous movement rep-resentations like splines, the learned pattern gener-ators remain autonomous dynamical systems whichrobustly cope with external perturbations that disruptthe time ow of the original pattern, and which canalso be modi ed on-line by additional perceptual vari-ables. A simple extension allows to cope with mul-tiple degrees-of-freedom (DOF) patterns, where allDOFs share the same fundamental frequency but,otherwise, can move in arbitrary phase and ampli-tude o sets to each other. We evaluate our meth-ods in learning from demonstration with an actual30 DOF humanoid robot. Figure-8 and drummingmovements are demonstrated by a human, recordedin joint angle space with an exoskeleton, and em-bedded in multi-dimensional rhythmic pattern gener-ators. The learned patterns can be used by the robotin various workspace locations and from arbitraryinitial conditions. Spatial and temporal invarianceof the pattern generators allow easy amplitude andspeed scaling without losing the qualitative signatureof a movement. This novel way of creating rhyth-mic patterns could tremendously facilitate rhythmicmovement generation, in particular in locomotion ofrobots and neural prosthetics in clinical applications.
---
paper_title: Imitating human playing styles in Super Mario Bros

paper_content:

Abstract We describe and compare several methods for generating game character controllers that mimic the playing style of a particular human player, or of a population of human players, across video game levels. Similarity in playing style is measured through an evaluation framework, that compares the play trace of one or several human players with the punctuated play trace of an AI player. The methods that are compared are either hand-coded, direct (based on supervised learning) or indirect (based on maximising a similarity measure). We find that a method based on neuroevolution performs best both in terms of the instrumental similarity measure and in phenomenological evaluation by human spectators. A version of the classic platform game “Super Mario Bros” is used as the testbed game in this study but the methods are applicable to other games that are based on character movement in space.
---
paper_title: Learning human arm movements by imitation: : Evaluation of a biologically inspired connectionist architecture

paper_content:

Abstract This paper evaluates a model of human imitation of abstract, two-arm movements. The model consists of a hierarchy of artificial neural networks, which are abstractions of brain regions involved in visuo-motor control. The model is validated in a biomechanical simulation of a 37 degree of freedom (DOF) humanoid. Input to the model are data from human arm movements recorded using video and marker-based tracking systems. Results show a high qualitative and quantitative agreement with human data. The model’s reproduction is better or comparable to that of human subjects imitating the same movements.
---
paper_title: Socially Assistive Robots in Elderly Care: A Systematic Review into Effects and Effectiveness

paper_content:

Abstract The ongoing development of robotics on the one hand and, on the other hand, the foreseen relative growth in number of elderly individuals suffering from dementia, raises the question of which contribution robotics could have to rationalize and maintain, or even improve the quality of care. The objective of this review was to assess the published effects and effectiveness of robot interventions aiming at social assistance in elderly care. We searched, using Medical Subject Headings terms and free words, in the CINAHL, MEDLINE, Cochrane, BIOMED, PUBMED, PsycINFO, and EMBASE databases. Also the IEEE Digital Library was searched. No limitations were applied for the date of publication. Only articles written in English were taken into account. Collected publications went through a selection process. In the first step, publications were collected from major databases using a search query. In the second step, 3 reviewers independently selected publications on their title, using predefined selection criteria. In the third step, publications were judged based on their abstracts by the same reviewers, using the same selection criteria. In the fourth step, one reviewer made the final selection of publications based on complete content. Finally, 41 publications were included in the review, describing 17 studies involving 4 robot systems. Most studies reported positive effects of companion-type robots on (socio)psychological (eg, mood, loneliness, and social connections and communication) and physiological (eg, stress reduction) parameters. The methodological quality of the studies was, mostly, low. Although positive effects were reported, the scientific value of the evidence was limited. The positive results described, however, prompt further effectiveness research in this field.
---
paper_title: Measuring Intelligence through Games

paper_content:

Abstract Artiﬁcial general intelligence (AGI) refers to research aimed at tackling the full problemof artiﬁcial intelligence, that is, create truly intelligent agents. This sets it apart from mostAI research which aims at solving relatively narrow domains, such as character recognition,motion planning, or increasing player satisfaction in games. But how do we know when anagent is truly intelligent? A common point of reference in the AGI community is Legg andHutter’s formal deﬁnition of universal intelligence, which has the appeal of simplicity andgenerality but is unfortunately incomputable.Games of various kinds are commonly used as benchmarks for “narrow” AI research,as they are considered to have many important properties. We argue that many of theseproperties carry over to the testing of general intelligence as well. We then sketch how suchtesting could practically be carried out. The central part of this sketch is an extension ofuniversal intelligence to deal with ﬁnite time, and the use of sampling of the space of gamesexpressed in a suitably biased game description language.Keywords: measure of intelligence, games
---
paper_title: Physical Human-Robot Interaction: Mutual Learning and Adaptation

paper_content:

Close physical interaction between robots and humans is a particularly challenging aspect of robot development. For successful interaction and cooperation, the robot must have the ability to adapt its behavior to the human counterpart. Based on our earlier work, we present and evaluate a computationally efficient machine learning algorithm that is well suited for such close-contact interaction scenarios. We show that this algorithm helps to improve the quality of the interaction between a robot and a human caregiver. To this end, we present two human-in-the-loop learning scenarios that are inspired by human parenting behavior, namely, an assisted standing-up task and an assisted walking task.
---
paper_title: A System for Robotic Heart Surgery that Learns to Tie Knots Using Recurrent Neural Networks

paper_content:

Tying suture knots is a time-consuming task performed frequently during minimally invasive surgery (MIS). Automating this task could greatly reduce total surgery time for patients. Current solutions to this problem replay manually programmed trajectories, but a more general and robust approach is to use supervised machine learning to smooth surgeon-given training trajectories and generalize from them. Since knottying generally requires a controller with internal memory to distinguish between identical inputs that require different actions at different points along a trajectory, it would be impossible to teach the system using traditional feedforward neural nets or support vector machines. Instead we exploit more powerful, recurrent neural networks (RNNs) with adaptive internal states. Results obtained using LSTM RNNs trained by the recent Evolino algorithm show that this approach can significantly increase the efficiency of suture knot tying in MIS over preprogrammed control
---
paper_title: Imitation learning through games: theory, implementation and evaluation

paper_content:

Despite a history of games-based research, academia has generally regarded ::: commercial games as a distraction from the serious business of AI, rather than as an ::: opportunity to leverage this existing domain to the advancement of our knowledge. ::: Similarly, the computer game industry still relies on techniques that were developed ::: several decades ago, and has shown little interest in adopting more progressive ::: academic approaches. In recent times, however, these attitudes have begun to change; ::: under- and post-graduate games development courses are increasingly common, ::: while the industry itself is slowly but surely beginning to recognise the potential ::: offered by modern machine-learning approaches, though games which actually ::: implement said approaches on more than a token scale remain scarce. ::: One area which has not yet received much attention from either academia or industry ::: is imitation learning, which seeks to expedite the learning process by exploiting data ::: harvested from demonstrations of a given task. While substantial work has been done ::: in developing imitation techniques for humanoid robot movement, there has been ::: very little exploration of the challenges posed by interactive computer games. Given ::: that such games generally encode reasoning and decision-making behaviours which ::: are inherently more complex and potentially more interesting than limb motion data, ::: that they often provide inbuilt facilities for recording human play, that the generation ::: and collection of training samples is therefore far easier than in robotics, and that ::: many games have vast pre-existing libraries of these recorded demonstrations, it is ::: fair to say that computer games represent an extremely fertile domain for imitation ::: learning research. ::: In this thesis, we argue in favour of using modern, commercial computer games to ::: study, model and reproduce humanlike behaviour. We provide an overview of the ::: biological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based ::: imitation. We then proceed to describe our contributions to the field of imitation ::: learning itself, which encompass three distinct categories: theory, implementation ::: and evaluation. ::: We first describe the development of a fully-featured Java API - the Quake2 Agent ::: Simulation Environment (QASE) - designed to facilitate both research and education ::: in imitation and general machine-learning, using the game Quake 2 as a testbed. We ::: outline our motivation for developing QASE, discussing the shortcomings of existing ::: APIs and the steps which we have taken to circumvent them. We describe QASE’s ::: network layer, which acts as an interface between the local AI routines and the ::: Quake 2 server on which the game environment is maintained, before detailing the ::: API’s agent architecture, which includes an interface to the MatLab programming ::: environment and the ability to parse and analyse full recordings of game sessions. ::: We conclude the chapter with a discussion of QASE’s adoption by numerous ::: universities as both an undergraduate teaching tool and research platform. ::: We then proceed to describe the various imitative mechanisms which we have ::: developed using QASE and its MatLab integration facilities. We first outline a ::: behaviour model based on a well-known psychological model of human planning. ::: Drawing upon previous research, we also identify a set of believability criteria - ::: elements of agent behaviour which are of particular importance in determining the ::: “humanness” of its in-game appearance. We then detail a reinforcement-learning ::: approach to imitating the human player’s navigation of his environment, centred ::: upon his pursuit of items as strategic goals. In the subsequent section, we describe ::: the integration of this strategic system with a Bayesian mechanism for the imitation ::: of tactical and motion-modelling behaviours. Finally, we outline a model for the ::: imitation of reactive combat behaviours; specifically, weapon-selection and aiming. Experiments are presented in each case to demonstrate the imitative mechanisms’ ::: ability to accurately reproduce observed behaviours. ::: Finally, we criticise the lack of any existing methodology to formally gauge the ::: believability of game agents, and observe that the few previous attempts have been ::: extremely ad-hoc and informal. We therefore propose a generalised approach to such ::: testing; the Bot-Oriented Turing Test (BOTT). This takes the form of an anonymous ::: online questionnaire, an accompanying protocol to which examiners should adhere, ::: and the formulation of a believability index which numerically expresses each agent’s ::: humanness as indicated by its observers, weighted by their experience and the ::: accuracy with which the agents were identified. To both validate the survey approach ::: and to determine the efficacy of our imitative models, we present a series of ::: experiments which use the believability test to evaluate our own imitation agents ::: against both human players and traditional artificial bots. We demonstrate that our ::: imitation agents perform substantially better than even a highly-regarded rule-based ::: agent, and indeed approach the believability of actual human players. ::: Some suggestions for future directions in our research, as well as a broader ::: discussion of open questions, conclude this thesis.
---
paper_title: Learning Two-Person Interaction Models for Responsive Synthetic Humanoids

paper_content:

JVRB, 11(2014), no. 1. - Imitation learning is a promising approach for generating life-like behaviors of virtual humans and humanoid robots. So far, however, imitation learning has been mostly restricted to single agent settings where observed motions are adapted to new environment conditions but not to the dynamic behavior of interaction partners. In this paper, we introduce a new imitation learning approach that is based on the simultaneous motion capture of two human interaction partners. From the observed interactions, low-dimensional motion models are extracted and a mapping between these motion models is learned. This interaction model allows the real-time generation of agent behaviors that are responsive to the body movements of an interaction partner. The interaction model can be applied both to the animation of virtual characters as well as to the behavior generation for humanoid robots.
---
paper_title: Controller for TORCS created by imitation

paper_content:

This paper is an initial approach to create a controller for the game TORCS by learning how another controller or humans play the game. We used data obtained from two controllers and from one human player. The first controller is the winner of the WCCI 2008 Simulated Car Racing Competition, and the second one is a hand coded controller that performs a complete lap in all tracks. First, each kind of controller is imitated separately, then a mix of the data is used to create new controllers. The imitation is performed by means of training a feed forward neural network with the data, using the backpropagation algorithm for learning.
---
paper_title: Programming full-body movements for humanoid robots by observation

paper_content:

Abstract The formulation and optimization of joint trajectories for humanoid robots is quite different from this same task for standard robots because of the complexity of humanoid robots’ kinematics and dynamics. In this paper we exploit the similarity between human motion and humanoid robot motion to generate joint trajectories for humanoids. In particular, we show how to transform human motion information captured by an optical tracking device into a high dimensional trajectory for a humanoid robot. We propose an automatic approach to relate humanoid robot kinematic parameters to the kinematic parameters of a human performer. Based on this relationship we infer the desired trajectories in robot joint space. B-spline wavelets are utilized to efficiently represent the trajectories. The density of the basis functions on the time axis is selected automatically. Large-scale optimization techniques are employed to solve the underlying computational problems efficiently. We applied our method to the task of teaching a humanoid robot how to make various naturally looking movements.
---
paper_title: Defining socially assistive robotics

paper_content:

This paper defines the research area of socially assistive robotics, focusing on assisting people through social interaction. While much attention has been paid to robots that provide assistance to people through physical contact (which we call contact assistive robotics), and to robots that entertain through social interaction (social interactive robotics), so far there is no clear definition of socially assistive robotics. We summarize active social assistive research projects and classify them by target populations, application domains, and interaction methods. While distinguishing these from socially interactive robotics endeavors, we discuss challenges and opportunities that are specific to the growing field of socially assistive robotics.
---
paper_title: Teaching robots by moulding behavior and scaffolding the environment

paper_content:

Programming robots to carry out useful tasks is both a complex and non-trivial exercise. A simple and intuitive method to allow humans to train and shape robot behaviour is clearly a key goal in making this task easier. This paper describes an approach to this problem based on studies of social animals where two teaching strategies are applied to allow a human teacher to train a robot by moulding its actions within a carefully scaffolded environment. Within these enviroments sets of competences can be built by building stateslash action memory maps of the robot's interaction within that environment. These memory maps are then polled using a k-nearest neighbour based algorithm to provide a generalised competence. We take a novel approach in building the memory models by allowing the human teacher to construct them in a hierarchical manner. This mechanism allows a human trainer to build and extend an action-selection mechanism into which new skills can be added to the robot's repertoire of existing competencies. These techniques are implemented on physical Khepera miniature robots and validated on a variety of tasks.
---
paper_title: Learning by observation with mobile robots: a computational approach

paper_content:

We present a computational approach to learning by observation (LBO) that allows users to program mobile robots by demonstrating a task. Unlike previous approaches, our system incorporates statistical-learning techniques and concepts from control theory to reduce the amount of domain knowledge needed to infer the intent of the user. To improve the generalization ability of the system, the user can demonstrate the task multiple times. We extract task subgoals from these demonstrations and automatically associate them with objects in the environment. As these objects move, the subgoals are updated accordingly. This gives our system the ability to learn from demonstrations performed in different environments. In this paper, we present the concepts used in our LBO system as well as experimental laboratory results in learning motor-skill tasks.
---
paper_title: Probabilistic Gaze Imitation and Saliency Learning in a Robotic Head

paper_content:

Imitation is a powerful mechanism for transferring knowledge from an instructor to a naive observer, one that is deeply contingent on a state of shared attention between these two agents. In this paper we present Bayesian algorithms that implement the core of an imitation learning framework. We use gaze imitation, coupled with task-dependent saliency learning, to build a state of shared attention between the instructor and observer. We demonstrate the performance of our algorithms in a gaze following and saliency learning task implemented on an active vision robotic head. Our results suggest that the ability to follow gaze and learn instructor-and task-specific saliency models could play a crucial role in building systems capable of complex forms of human-robot interaction.
---
paper_title: Schema design and implementation of the grasp-related mirror neuron system

paper_content:

Mirror neurons within a monkey's premotor area F5 fire not only when the monkey performs a certain class of actions but also when the monkey observes another monkey (or the experimenter) perform a similar action. It has thus been argued that these neurons are crucial for understanding of actions by others. We offer the hand-state hypothesis as a new explanation of the evolution of this capability: the basic functionality of the F5 mirror system is to elaborate the appropriate feedback – what we call the hand state– for opposition-space based control of manual grasping of an object. Given this functionality, the social role of the F5 mirror system in understanding the actions of others may be seen as an exaptation gained by generalizing from one's own hand to an other's hand. In other words, mirror neurons first evolved to augment the “canonical” F5 neurons (active during self-movement based on observation of an object) by providing visual feedback on “hand state,” relating the shape of the hand to the shape of the object. We then introduce the MNS1 (mirror neuron system 1) model of F5 and related brain regions. The existing Fagg–Arbib–Rizzolatti–Sakata model represents circuitry for visually guided grasping of objects, linking the anterior intraparietal area (AIP) with F5 canonical neurons. The MNS1 model extends the AIP visual pathway by also modeling pathways, directed toward F5 mirror neurons, which match arm–hand trajectories to the affordances and location of a potential target object. We present the basic schemas for the MNS1 model, then aggregate them into three “grand schemas”– visual analysis of hand state, reach and grasp, and the core mirror circuit – for each of which we present a useful implementation (a non-neural visual processing system, a multijoint 3-D kinematics simulator, and a learning neural network, respectively). With this implementation we show how the mirror system may learnto recognize actions already in the repertoire of the F5 canonical neurons. We show that the connectivity pattern of mirror neuron circuitry can be established through training, and that the resultant network can exhibit a range of novel, physiologically interesting behaviors during the process of action recognition. We train the system on the basis of final grasp but then observe the whole time course of mirror neuron activity, yielding predictions for neurophysiological experiments under conditions of spatial perturbation, altered kinematics, and ambiguous grasp execution which highlight the importance of the timingof mirror neuron activity.
---
paper_title: Learning Collaborative Impedance-based Robot Behaviors ∗

paper_content:

Research in learning from demonstration has focused on transferring movements from humans to robots. However, a need is arising for robots that do not just replicate the task on their own, but that also interact with humans in a safe and natural way to accomplish tasks cooperatively. Robots with variable impedance capabilities opens the door to new challenging applications, where the learning algorithms must be extended by encapsulating force and vision information. In this paper we propose a framework to transfer impedance-based behaviors to a torque-controlled robot by kinesthetic teaching. The proposed model encodes the examples as a task-parameterized statistical dynamical system, where the robot impedance is shaped by estimating virtual stiffness matrices from the set of demonstrations. A collaborative assembly task is used as testbed. The results show that the model can be used to modify the robot impedance along task execution to facilitate the collaboration, by triggering stiff and compliant behaviors in an on-line manner to adapt to the user's actions.
---

Title: Imitation Learning: A Survey of Learning Methods

Section 1: INTRODUCTION
Description 1: Introduce the field of imitation learning, its significance, and the motivation behind the survey. Discuss advancements, applications, and the importance of human-like behavior in AI.

Section 2: PROBLEM FORMULATION
Description 2: Formalize the problem of imitation learning, providing definitions and preliminaries. Discuss the process of learning policies from demonstrations and experience, and introduce Markov decision processes.

Section 3: FEATURE REPRESENTATIONS
Description 3: Explore methods for representing observable states as feature vectors. Discuss raw features, manually designed features, and extracted features, and their roles in learning policies.

Section 4: HANDLING DEMONSTRATIONS
Description 4: Discuss challenges in handling demonstrations, including the correspondence problem and sensor errors. Describe methods to create mappings and address incomplete or inaccurate demonstrations.

Section 5: LEARNING MOTION
Description 5: Review methods for directly learning policies from demonstrations. Cover supervised learning techniques, classification, regression, and hierarchical models for learning motor actions.

Section 6: INDIRECT LEARNING
Description 6: Examine methods for refining learned policies, including reinforcement learning, optimization approaches, and transfer learning. Discuss reinforcement learning’s role in improving and complementing imitation learning.

Section 7: MULTIAGENT IMITATION
Description 7: Present methods for incorporating imitation learning in multiagent systems. Discuss the complexities of multiagent interactions and the benefits of learning from observation and cooperation.

Section 8: EVALUATION
Description 8: Describe methods for evaluating imitation learning algorithms, including quantitative and qualitative approaches. Discuss challenges in standardizing evaluation and the importance of benchmarking.

Section 9: APPLICATIONS
Description 9: Highlight potential applications of imitation learning, including autonomous vehicles, assistive robotics, electronic games, humanoid robots, and industrial automation. Discuss the current state and future potential of these applications.

Section 10: CONCLUSION AND FUTURE DIRECTIONS
Description 10: Summarize the survey and discuss the key challenges and directions for future research. Emphasize the importance of generalization, benchmarking, and multiagent learning in advancing the field of imitation learning.