Abstraction Levels for Robotic Imitation: Overview and Computational Approaches
---
paper_title: Robot See, Robot Do : An Overview of Robot Imitation

paper_content:

A belt grinding machine which is equipped with an elastically flexible load beam extending over the entire operative length of an endless grinding belt for applying the necessary grinding pressure to the belt. The effective pressure-applying surface of said beam is suspended for elevational control from support elements distributed at intervals along the length of the beam and attached to an elevationally movable carrier preferably in the form of transverse beams mounted on push rods operable by a pressure medium. The push rods are associated with constant delay mechanism adjustable to the difference in time between the instant of response of a sensing element to the passage of the leading end of a workpiece and the instant the grinding action begins, on the one hand, and between the instant of response of a sensing element to the passage of the trailing end of the workpiece and the instant the grinding action terminates, on the other hand.
---
paper_title: Computational Approaches to Motor Learning by Imitation

paper_content:

Movement imitation requires a complex set of mechanisms that map an observed movement of a teacher onto one's own movement apparatus. Relevant problems include movement recognition, pose estimation, pose tracking, body correspondence, coordinate transformation from external to egocentric space, matching of observed against previously learned movement, resolution of redundant degrees-of-freedom that are unconstrained by the observation, suitable movement representations for imitation, modularization of motor control, etc. All of these topics by themselves are active research problems in computational and neurobiological sciences, such that their combination into a complete imitation system remains a daunting undertaking-indeed, one could argue that we need to understand the complete perception-action loop. As a strategy to untangle the complexity of imitation, this paper will examine imitation purely from a computational point of view, i.e. we will review statistical and mathematical approaches that have been suggested for tackling parts of the imitation problem, and discuss their merits, disadvantages and underlying principles. Given the focus on action recognition of other contributions in this special issue, this paper will primarily emphasize the motor side of imitation, assuming that a perceptual system has already identified important features of a demonstrated movement and created their corresponding spatial information. Based on the formalization of motor control in terms of control policies and their associated performance criteria, useful taxonomies of imitation learning can be generated that clarify different approaches and future research directions.
---
paper_title: Affordance-based imitation learning in robots

paper_content:

In this paper we build an imitation learning algorithm for a humanoid robot on top of a general world model provided by learned object affordances. We consider that the robot has previously learned a task independent affordance-based model of its interaction with the world. This model is used to recognize the demonstration by another agent (a human) and infer the task to be learned. We discuss several important problems that arise in this combined framework, such as the influence of an inaccurate model in the recognition of the demonstration. We illustrate the ideas in the paper with some experimental results obtained with a real robot.
---
paper_title: A survey of robot learning from demonstration

paper_content:

We present a comprehensive survey of robot Learning from Demonstration (LfD), a technique that develops policies from example state to action mappings. We introduce the LfD design choices in terms of demonstrator, problem space, policy derivation and performance, and contribute the foundations for a structure in which to categorize LfD research. Specifically, we analyze and categorize the multiple ways in which examples are gathered, ranging from teleoperation to imitation, as well as the various techniques for policy derivation, including matching functions, dynamics models and plans. To conclude we discuss LfD limitations and related promising areas for future research.
---
paper_title: A Developmental Roadmap for Learning by Imitation in Robots

paper_content:

In this paper, we present a strategy whereby a robot acquires the capability to learn by imitation following a developmental pathway consisting on three levels: 1) sensory-motor coordination; 2) world interaction; and 3) imitation. With these stages, the system is able to learn tasks by imitating human demonstrators. We describe results of the different developmental stages, involving perceptual and motor skills, implemented in our humanoid robot, Baltazar. At each stage, the system's attention is drawn toward different entities: its own body and, later on, objects and people. Our main contributions are the general architecture and the implementation of all the necessary modules until imitation capabilities are eventually acquired by the robot. Also, several other contributions are made at each level: learning of sensory-motor maps for redundant robots, a novel method for learning how to grasp objects, and a framework for learning task description from observation for program-level imitation. Finally, vision is used extensively as the sole sensing modality (sometimes in a simplified setting) avoiding the need for special data-acquisition hardware
---
paper_title: A MODEL OF IMITATION LEARNING OF ALGORITHMS FROM WORKED EXAMPLES

paper_content:

Imitation is an important learning mechanism of widespread utility and common occurrence. This article presents a theory and working computational model of the detailed mechanisms of imitation. The model is in the restricted domain of the learning of pencil and paper procedures. The task that is modelled is of a teacher demonstrating the steps of a procedure, such as long division to a student by means of one or more examples. Such a task can be learned by an imitation-learning mechanism, but the mechanism has a much wider range of application. Imitation is treated as a four-stage process: the events performed by the teacher are segmented by the learner; the events are encoded and explained in terms of spatial relations between objects; repeated patterns in the events are recognized; and finally, different examples are merged together. This model is implemented as a computer program learning algorithms from worked examples (LAWE).
---
paper_title: Learning by Watching: Extracting Reusable Task Knowledge from Visual Observation of Human Performance

paper_content:

A novel task instruction method for future intelligent robots is presented, In our method, a robot learns reusable task plans by watching a human perform assembly tasks. Functional units and working algorithms for visual recognition and analysis of human action sequences are presented. The overall system is model based and integrated at the symbolic level. Temporal segmentation of a continuous task performance into meaningful units and identification of each operation is processed in real time by concurrent recognition processes under active attention control. Dependency among assembly operations in the recognized action sequence is analyzed, which results in a hierarchical task plan describing the higher level structure of the task. In another workspace with a different initial state, the system re-instantiates and executes the task plan to accomplish an equivalent goal. The effectiveness of our method is supported by experimental results with block assembly tasks. >
---
paper_title: Implicit Imitation in Multiagent Reinforcement Learning

paper_content:

Imitation is actively being studied as an effective means of learning in multi-agent environments. It allows an agent to learn how to act well (perhaps optimally) by passively observing the actions of cooperative teachers or other more experienced agents its environment. We propose a straightforward imitation mechanism called model extraction that can be integrated easily into standard model-based reinforcement learning algorithms. Roughly, by observing a mentor with similar capabilities, an agent can extract information about its own capabilities in unvisited parts of state space. The extracted information can accelerate learning dramatically. We illustrate the benefits of model extraction by integrating it with prioritized sweeping, and demonstrating improved performance and convergence through observation of single and multiple mentors. Though we make some stringent assumptions regarding observability, possible interactions and common abilities, we briefly comment on extensions of the model that relax these.
---
paper_title: Accelerating Reinforcement Learning through Implicit Imitation

paper_content:

Imitation can be viewed as a means of enhancing learning in multiagent environments. It augments an agent's ability to learn useful behaviors by making intelligent use of the knowledge implicit in behaviors demonstrated by cooperative teachers or other more experienced agents. We propose and study a formal model of implicit imitation that can accelerate reinforcement learning dramatically in certain cases. Roughly, by observing a mentor, a reinforcement-learning agent can extract information about its own capabilities in, and the relative value of, unvisited parts of the state space. We study two specific instantiations of this model, one in which the learning agent and the mentor have identical abilities, and one designed to deal with agents and mentors with difierent action sets. We illustrate the benefits of implicit imitation by integrating it with prioritized sweeping, and demonstrating improved performance and convergence through observation of single and multiple mentors. Though we make some stringent assumptions regarding observability and possible interactions, we briefly comment on extensions of the model that relax these restricitions.
---
paper_title: Bayesian imitation learning in game characters

paper_content:

Imitation learning is a powerful mechanism applied by primates and humans. It allows for a straightforward acquisition of behaviours that, through observation, are known to solve everyday tasks. Recently, a Bayesian formulation has been proposed that provides a mathematical model of imitation learning. In this paper, we apply this framework to the problem of programming believable computer games characters. We will present experiments in imitation learning from the network traffic of multi-player online games. Our results underline that this indeed produces agents that behave more human-like than characters controlled by common game AI techniques.
---
paper_title: Hidden Markov Model Approach to Skill Learning and Its Application to Telerobotics

paper_content:

In this paper, we discuss the problem of how human skill can be represented as a parametric model using a hidden Markov model (HMM), and how an HMM-based skill model can be used to learn human skill. HMM is feasible to characterize a doubly stochastic process--measurable action and immeasurable mental states--that is involved in the skill learning. We formulated the learning problem as a multidimensional HMM and developed a testbed for a variety of skill learning applications. Based on "the most likely performance" criterion, the best action sequence can be selected from all previously measured action data by modeling the skill as an HMM. The proposed method has been implemented in the teleoperation control of space station robot system, and some important implementation issues have been discussed. The method allows a robot to learn human skill in certain tasks and to improve motion performance.
---
paper_title: Natural Actor-Critic

paper_content:

In this paper, we suggest a novel reinforcement learning architecture, the Natural Actor-Critic. The actor updates are achieved using stochastic policy gradients employing Amari's natural gradient approach, while the critic obtains both the natural policy gradient and additional parameters of a value function simultaneously by linear regression. We show that actor improvements with natural policy gradients are particularly appealing as these are independent of coordinate frame of the chosen policy representation, and can be estimated more efficiently than regular policy gradients. The critic makes use of a special basis function parameterization motivated by the policy-gradient compatible function approximation. We show that several well-known reinforcement learning methods such as the original Actor-Critic and Bradtke's Linear Quadratic Q-Learning are in fact Natural Actor-Critic algorithms. Empirical evaluations illustrate the effectiveness of our techniques in comparison to previous methods, and also demonstrate their applicability for learning control on an anthropomorphic robot arm.
---
paper_title: Is imitation learning the route to humanoid robots?

paper_content:

Summary This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It will be postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in form of movement primitives. It will be reviewed how research on representations of, and functional connections between action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning will also be described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical learning research. Parallels and differences between biological and computational approaches to imitation will be highlighted and an overview of current projects that actually employ imitation learning for humanoid robots will be given.
---
paper_title: A Developmental Roadmap for Learning by Imitation in Robots

paper_content:

In this paper, we present a strategy whereby a robot acquires the capability to learn by imitation following a developmental pathway consisting on three levels: 1) sensory-motor coordination; 2) world interaction; and 3) imitation. With these stages, the system is able to learn tasks by imitating human demonstrators. We describe results of the different developmental stages, involving perceptual and motor skills, implemented in our humanoid robot, Baltazar. At each stage, the system's attention is drawn toward different entities: its own body and, later on, objects and people. Our main contributions are the general architecture and the implementation of all the necessary modules until imitation capabilities are eventually acquired by the robot. Also, several other contributions are made at each level: learning of sensory-motor maps for redundant robots, a novel method for learning how to grasp objects, and a framework for learning task description from observation for program-level imitation. Finally, vision is used extensively as the sole sensing modality (sometimes in a simplified setting) avoiding the need for special data-acquisition hardware
---
paper_title: Teaching from examples in assembly and manipulation of snack food ingredients by robot

paper_content:

We report the results of a programme of work that allows us to specify and execute robotic assembly of snack food products such as sandwiches and pizzas simply by presenting to the system hand-prepared examples of the desired results of the robot operations. We summarise a detailed experimental evaluation of our pick and place work, in which placement location and orientation are taught by example. We also describe and evaluate our recent work on automated spreading of ingredients so as to form specific surface profiles. Our methods facilitate rapid reconfiguration to accommodate new product variants without the need for reprogramming.
---
paper_title: A unified framework for imitation-like behaviors

paper_content:

In this paper, we combine the formal methods from re- inforcement learning with the paradigm of imitation learning. The extension of the reinforcement learning framework to integrate the information provided by an expert (demonstrator) has the important advantage of allowing a clear decrease of the time necessary to learn certain robotic tasks. Hence, learning by imitation can be interpreted as a mechanism for fast skill transfer. Another contribution of this paper consists in showing that our formalism is able to model dif- ferent types of imitation-learning that are described in the biological literature. It thus unifies in the same abstract model what used to be addressed as separate behavioral patterns. We illustrate the applica- tion of these methods in simulation and with a real robot.
---
paper_title: Social Learning Mechanisms Compared in a Simple Environment

paper_content:

Social learning can be adaptive, but little is known about the underlying mechanisms. Many researchers have focused on imitation but this may have led to simpler mechanisms being underestimated. We demonstrate in simulation that imitative learning is not always the best strategy for a group-living animal, and that the effectiveness of any such strategy will depend on details of the environment and the animal's lifestyle. We show that observations of behavioural convergence or "traditions" might suggest effective social learning, but are meaningless considered alone.
---
paper_title: Imitation in Animals: Evidence, Function, and Mechanisms

paper_content:

The terms sociallearning and social influence have been used descriptively and theoretically to characterize a broad range of animal behavior from physical antipredatory adaptations such as eye spots, which are totally under genetic control, to the human capacity for the exaggeration of individual characteristics, known as caricature, which are largely under cognitive control. In the present review, the various forms of social influence and social learning are identified and distinghished from imitation, a term that generally has been reserved for behavioral matching that cannot be accounted for using simpler specifically predisposed, motivational, or learning mechanisms. It is suggested that much of the ambiguity in the literature concerning the various forms of social learning can be attributed to the distinction between the function of a behavior and the mechanisms responsible for its occurrence. Finally, the various mechanisms that have been proposed to account for imitative learning are presented and...
---
paper_title: Is imitation learning the route to humanoid robots?

paper_content:

Summary This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It will be postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in form of movement primitives. It will be reviewed how research on representations of, and functional connections between action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning will also be described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical learning research. Parallels and differences between biological and computational approaches to imitation will be highlighted and an overview of current projects that actually employ imitation learning for humanoid robots will be given.
---
paper_title: A Developmental Roadmap for Learning by Imitation in Robots

paper_content:

In this paper, we present a strategy whereby a robot acquires the capability to learn by imitation following a developmental pathway consisting on three levels: 1) sensory-motor coordination; 2) world interaction; and 3) imitation. With these stages, the system is able to learn tasks by imitating human demonstrators. We describe results of the different developmental stages, involving perceptual and motor skills, implemented in our humanoid robot, Baltazar. At each stage, the system's attention is drawn toward different entities: its own body and, later on, objects and people. Our main contributions are the general architecture and the implementation of all the necessary modules until imitation capabilities are eventually acquired by the robot. Also, several other contributions are made at each level: learning of sensory-motor maps for redundant robots, a novel method for learning how to grasp objects, and a framework for learning task description from observation for program-level imitation. Finally, vision is used extensively as the sole sensing modality (sometimes in a simplified setting) avoiding the need for special data-acquisition hardware
---
paper_title: How do apes ape?

paper_content:

In the wake of telling critiques of the foundations on which earlier conclusions were based, the last 15 years have witnessed a renaissance in the study of social learning in apes. As a result, we are able to review 31 experimental studies from this period in which social learning in chimpanzees, gorillas, and orangutans has been investigated. The principal question framed at the beginning of this era, Do apes ape? has been answered in the affirmative, at least in certain conditions. The more interesting question now is, thus, How do apes ape? Answering this question has engendered richer taxonomies of the range of social-learning processes at work and new methodologies to uncover them. Together, these studies suggest that apes ape by employing a portfolio of alternative social-learning processes in flexibly adaptive ways, in conjunction with nonsocial learning. We conclude by sketching the kind of decision tree that appears to underlie the deployment of these alternatives.
---
paper_title: The motor theory of speech perception revised

paper_content:

A motor theory of speech perception, initially proposed to account for results of early experiments with synthetic speech, is now extensively revised to accommodate recent findings, and to relate the assumptions of the theory to those that might be made about other perceptual modes. According to the revised theory, phonetic information is perceived in a biologically distinct system, a ‘module’ specialized to detect the intended gestures of the speaker that are the basis for phonetic categories. Built into the structure of this module is the unique but lawful relationship between the gestures and the acoustic patterns in which they are variously overlapped. In consequence, the module causes perception of phonetic structure without translation from preliminary auditory impressions. Thus, it is comparable to such other modules as the one that enables an animal to localize sound. Peculiar to the phonetic module are the relation between perception and production it incorporates and the fact that it must compete with other modules for the same stimulus variations.
---
paper_title: Investigating Action Understanding: Inferential Processes versus Action Simulation

paper_content:

In our daily life, we continuously monitor others' behaviors and interpret them in terms of goals, intentions, and reasons. Despite their central importance for predicting and interpreting each other's actions, the functional mechanisms and neural circuits involved in action understanding remain highly controversial. Two alternative accounts have been advanced. Simulation theory assumes that we understand actions by simulating the observed behavior through a direct matching process that activates the mirror-neuron circuit. The alternative interpretive account assumes that action understanding is based on specialized inferential processes activating brain areas with no mirror properties. Although both approaches recognize the central role of contextual information in specifying action intentions, their respective accounts of this process differ in significant respects. Here, we investigated the role of context in action understanding by using functional brain imaging while participants observed an unusual action in implausible versus plausible contexts. We show that brain areas that are part of a network involved in inferential interpretive processes of rationalization and mentalization but that lack mirror properties are more active when the action occurs in an implausible context. However, no differential activation was found in the mirror network. Our findings support the assumption that action understanding in novel situations is primarily mediated by an inferential interpretive system rather than the mirror system.
---
paper_title: The motor theory of speech perception reviewed

paper_content:

More than 50 years after the appearance of the motor theory of speech perception, it is timely to evaluate its three main claims that (1) speech processing is special, (2) perceiving speech is perceiving gestures, and (3) the motor system is recruited for perceiving speech. We argue that to the extent that it can be evaluated, the first claim is likely false. As for the second claim, we review findings that support it and argue that although each of these findings may be explained by alternative accounts, the claim provides a single coherent account. As for the third claim, we review findings in the literature that support it at different levels of generality and argue that the claim anticipated a theme that has become widespread in cognitive science.
---
paper_title: Multiple paired forward and inverse models for motor control

paper_content:

Abstract Humans demonstrate a remarkable ability to generate accurate and appropriate motor behavior under many different and often uncertain environmental conditions. In this paper, we propose a modular approach to such motor learning and control. We review the behavioral evidence and benefits of modularity, and propose a new architecture based on multiple pairs of inverse (controller) and forward (predictor) models. Within each pair, the inverse and forward models are tightly coupled both during their acquisition, through motor learning, and use, during which the forward models determine the contribution of each inverse model's output to the final motor command. This architecture can simultaneously learn the multiple inverse models necessary for control as well as how to select the inverse models appropriate for a given environment. Finally, we describe specific predictions of the model, which can be tested experimentally.
---
paper_title: Mirror neurons and imitation: a computationally guided review.

paper_content:

Neurophysiology reveals the properties of individual mirror neurons in the macaque while brain imaging reveals the presence of 'mirror systems' (not individual neurons) in the human. Current conceptual models attribute high level functions such as action understanding, imitation, and language to mirror neurons. However, only the first of these three functions is well-developed in monkeys. We thus distinguish current opinions (conceptual models) on mirror neuron function from more detailed computational models. We assess the strengths and weaknesses of current computational models in addressing the data and speculations on mirror neurons (macaque) and mirror systems (human). In particular, our mirror neuron system (MNS), mental state inference (MSI) and modular selection and identification for control (MOSAIC) models are analyzed in more detail. Conceptual models often overlook the computational requirements for posited functions, while too many computational models adopt the erroneous hypothesis that mirror neurons are interchangeable with imitation ability. Our meta-analysis underlines the gap between conceptual and computational models and points out the research effort required from both sides to reduce this gap.
---
paper_title: Action recognition in the premotor cortex.

paper_content:

We recorded electrical activity from 532 neurons in the rostral part of inferior area 6 (area F5) of two macaque monkeys. Previous data had shown that neurons of this area discharge during goal-directed hand and mouth movements. We describe here the properties of a newly discovered set of F5 neurons (‘mirror neurons’, n = 92) all of which became active both when the monkey performed a given action and when it observed a similar action performed by the experimenter. Mirror neurons, in order to be visually triggered, required an interaction between the agent of the action and the object of it. The sight of the agent alone or of the object alone (three-dimensional objects, food) were ineffective. Hand and the mouth were by far the most effective agents. The actions most represented among those activating mirror neurons were grasping, manipulating and placing. In most mirror neurons (92%) there was a clear relation between the visual action they responded to and the motor response they coded. In ∼30% of mirror neurons the congruence was very strict and the effective observed and executed actions corresponded both in terms of general action (e.g. grasping) and in terms of the way in which that action was executed (e.g. precision grip). We conclude by proposing that mirror neurons form a system for matching observation and execution of motor actions. We discuss the possible role of this system in action recognition and, given the proposed homology between F5 and human Brocca’s region, we posit that a matching system, similar to that of mirror neurons exists in humans and could be involved in recognition of actions as well as phonetic gestures.
---
paper_title: The motor theory of speech perception revised

paper_content:

A motor theory of speech perception, initially proposed to account for results of early experiments with synthetic speech, is now extensively revised to accommodate recent findings, and to relate the assumptions of the theory to those that might be made about other perceptual modes. According to the revised theory, phonetic information is perceived in a biologically distinct system, a ‘module’ specialized to detect the intended gestures of the speaker that are the basis for phonetic categories. Built into the structure of this module is the unique but lawful relationship between the gestures and the acoustic patterns in which they are variously overlapped. In consequence, the module causes perception of phonetic structure without translation from preliminary auditory impressions. Thus, it is comparable to such other modules as the one that enables an animal to localize sound. Peculiar to the phonetic module are the relation between perception and production it incorporates and the fact that it must compete with other modules for the same stimulus variations.
---
paper_title: The motor theory of speech perception reviewed

paper_content:

More than 50 years after the appearance of the motor theory of speech perception, it is timely to evaluate its three main claims that (1) speech processing is special, (2) perceiving speech is perceiving gestures, and (3) the motor system is recruited for perceiving speech. We argue that to the extent that it can be evaluated, the first claim is likely false. As for the second claim, we review findings that support it and argue that although each of these findings may be explained by alternative accounts, the claim provides a single coherent account. As for the third claim, we review findings in the literature that support it at different levels of generality and argue that the claim anticipated a theme that has become widespread in cognitive science.
---
paper_title: Multiple paired forward and inverse models for motor control

paper_content:

Abstract Humans demonstrate a remarkable ability to generate accurate and appropriate motor behavior under many different and often uncertain environmental conditions. In this paper, we propose a modular approach to such motor learning and control. We review the behavioral evidence and benefits of modularity, and propose a new architecture based on multiple pairs of inverse (controller) and forward (predictor) models. Within each pair, the inverse and forward models are tightly coupled both during their acquisition, through motor learning, and use, during which the forward models determine the contribution of each inverse model's output to the final motor command. This architecture can simultaneously learn the multiple inverse models necessary for control as well as how to select the inverse models appropriate for a given environment. Finally, we describe specific predictions of the model, which can be tested experimentally.
---
paper_title: Mirror neurons and imitation: a computationally guided review.

paper_content:

Neurophysiology reveals the properties of individual mirror neurons in the macaque while brain imaging reveals the presence of 'mirror systems' (not individual neurons) in the human. Current conceptual models attribute high level functions such as action understanding, imitation, and language to mirror neurons. However, only the first of these three functions is well-developed in monkeys. We thus distinguish current opinions (conceptual models) on mirror neuron function from more detailed computational models. We assess the strengths and weaknesses of current computational models in addressing the data and speculations on mirror neurons (macaque) and mirror systems (human). In particular, our mirror neuron system (MNS), mental state inference (MSI) and modular selection and identification for control (MOSAIC) models are analyzed in more detail. Conceptual models often overlook the computational requirements for posited functions, while too many computational models adopt the erroneous hypothesis that mirror neurons are interchangeable with imitation ability. Our meta-analysis underlines the gap between conceptual and computational models and points out the research effort required from both sides to reduce this gap.
---
paper_title: Mirror neurons and imitation learning as the driving force behind "the great leap forward" in human evolution

paper_content:

The discovery of mirror neurons in the frontal lobes of monkeys, and their potential relevance to human brain evolution — which I speculate on in this essay — is the single most important unreported (or at least, unpublicized) story of the decade. I predict that mirror neurons will do for psychology what DNA did for biology: they will provide a unifying framework and help explain a host of mental abilities that have hitherto remained mysterious and inaccessible to experiments.
---
paper_title: Action recognition in the premotor cortex.

paper_content:

We recorded electrical activity from 532 neurons in the rostral part of inferior area 6 (area F5) of two macaque monkeys. Previous data had shown that neurons of this area discharge during goal-directed hand and mouth movements. We describe here the properties of a newly discovered set of F5 neurons (‘mirror neurons’, n = 92) all of which became active both when the monkey performed a given action and when it observed a similar action performed by the experimenter. Mirror neurons, in order to be visually triggered, required an interaction between the agent of the action and the object of it. The sight of the agent alone or of the object alone (three-dimensional objects, food) were ineffective. Hand and the mouth were by far the most effective agents. The actions most represented among those activating mirror neurons were grasping, manipulating and placing. In most mirror neurons (92%) there was a clear relation between the visual action they responded to and the motor response they coded. In ∼30% of mirror neurons the congruence was very strict and the effective observed and executed actions corresponded both in terms of general action (e.g. grasping) and in terms of the way in which that action was executed (e.g. precision grip). We conclude by proposing that mirror neurons form a system for matching observation and execution of motor actions. We discuss the possible role of this system in action recognition and, given the proposed homology between F5 and human Brocca’s region, we posit that a matching system, similar to that of mirror neurons exists in humans and could be involved in recognition of actions as well as phonetic gestures.
---
paper_title: Investigating Action Understanding: Inferential Processes versus Action Simulation

paper_content:

In our daily life, we continuously monitor others' behaviors and interpret them in terms of goals, intentions, and reasons. Despite their central importance for predicting and interpreting each other's actions, the functional mechanisms and neural circuits involved in action understanding remain highly controversial. Two alternative accounts have been advanced. Simulation theory assumes that we understand actions by simulating the observed behavior through a direct matching process that activates the mirror-neuron circuit. The alternative interpretive account assumes that action understanding is based on specialized inferential processes activating brain areas with no mirror properties. Although both approaches recognize the central role of contextual information in specifying action intentions, their respective accounts of this process differ in significant respects. Here, we investigated the role of context in action understanding by using functional brain imaging while participants observed an unusual action in implausible versus plausible contexts. We show that brain areas that are part of a network involved in inferential interpretive processes of rationalization and mentalization but that lack mirror properties are more active when the action occurs in an implausible context. However, no differential activation was found in the mirror network. Our findings support the assumption that action understanding in novel situations is primarily mediated by an inferential interpretive system rather than the mirror system.
---
paper_title: Mirror neurons and imitation: a computationally guided review.

paper_content:

Neurophysiology reveals the properties of individual mirror neurons in the macaque while brain imaging reveals the presence of 'mirror systems' (not individual neurons) in the human. Current conceptual models attribute high level functions such as action understanding, imitation, and language to mirror neurons. However, only the first of these three functions is well-developed in monkeys. We thus distinguish current opinions (conceptual models) on mirror neuron function from more detailed computational models. We assess the strengths and weaknesses of current computational models in addressing the data and speculations on mirror neurons (macaque) and mirror systems (human). In particular, our mirror neuron system (MNS), mental state inference (MSI) and modular selection and identification for control (MOSAIC) models are analyzed in more detail. Conceptual models often overlook the computational requirements for posited functions, while too many computational models adopt the erroneous hypothesis that mirror neurons are interchangeable with imitation ability. Our meta-analysis underlines the gap between conceptual and computational models and points out the research effort required from both sides to reduce this gap.
---
paper_title: Cortical mechanism for the visual guidance of hand grasping movements in the monkey: A reversible inactivation study

paper_content:

Summary Picking up an object requires two basic motor operations: reaching and grasping. Neurophysiological studies in monkeys have suggested that the visuomotor transformations necessary for these two operations are carried out by separate parietofrontal circuits and that, for grasping, a key role is played by a specific sector of the ventral premotor cortex: area F5. The aim of the present study was to test the validity of this hypothesis by reversibly inactivating area F5 in monkeys trained to grasp objects of different shape, size and orientation. In separate sessions, the hand field of the primary motor cortex (area F1 or area 4) was also reversibly inactivated. The results showed that after inactivation of area F5 buried in the bank of the arcuate sulcus (the F5 sector where visuomotor neurones responding to object presentation are located), the hand shaping preceding grasping was markedly impaired and the hand posture was not appropriate for the object size and shape. The monkeys were eventually able to grasp the objects, but
---
paper_title: Inferring the goals of a nonhuman agent

paper_content:

Johnson, Slaughter, and Carey [Dev. Sci. 1 (1998) 233.] used infants' ability to follow the ‘gaze’ of novel objects to claim that infants' recognition of mentalistic agents is not isomorphic with person recognition but rather based on a set of nonarbitrary object recognition cues including the presence of a face and the ability to interact contingently with other agents. The current studies extend these findings with data based on infant imitation and the production of communicative gestures. The first study replicated Meltzoff's [Dev. Psychol. 31 (1995) 838.] reenactment of goals paradigm with a novel, nonhuman agent. Fifteen-month-olds were found to reenact both the completed and uncompleted/unseen goals of a novel object that had a face and interacted contingently with the experimenter and infant. Concurrently, infants directed many communicative gestures at the object. A second study excluded the possibility that the communicative gestures apparently directed at the object in Study 1 were in fact imitations of the experimenter's own behavior. The suggestion that novel, nonhuman objects are capable of eliciting such divergent behaviors as gaze-following, goal reenactment, and communicative gestures from infants, supports the claim that all of these behaviors are mediated by a central conceptual notion of mentalistic being, at least by the ages studied, and that that concept is not isomorphic with the concept person.
---
paper_title: Imitation in Animals: Evidence, Function, and Mechanisms

paper_content:

The terms sociallearning and social influence have been used descriptively and theoretically to characterize a broad range of animal behavior from physical antipredatory adaptations such as eye spots, which are totally under genetic control, to the human capacity for the exaggeration of individual characteristics, known as caricature, which are largely under cognitive control. In the present review, the various forms of social influence and social learning are identified and distinghished from imitation, a term that generally has been reserved for behavioral matching that cannot be accounted for using simpler specifically predisposed, motivational, or learning mechanisms. It is suggested that much of the ambiguity in the literature concerning the various forms of social learning can be attributed to the distinction between the function of a behavior and the mechanisms responsible for its occurrence. Finally, the various mechanisms that have been proposed to account for imitative learning are presented and...
---
paper_title: Learning from Other People's Mistakes: Causal Understanding in Learning To Use a Tool.

paper_content:

Despite considerable debate about whether nonhuman primates learn to use tools via imitation, this type of learning by children has received surprisingly little attention. The findings of two studies that go some way toward filling this gap are reported here. Study 1 showed that when 2- and 3-year-old children (N = 68) were shown a correct solution to a tool-using task (which they could not solve spontaneously), all the children in both age groups managed at least a partial solution. When children were shown an incorrect solution followed by a correct solution, 2-year-olds again produced only a partial solution. By contrast, most 3-year-olds produced a full solution. Study 2 replicated this age change in a separate sample of children (N = 100) with a different tool-using task. Study 2 also showed that 3-year-olds benefit from observing an incorrect action when it can be contrasted with a correct action: they chose the more effective of the two actions. Taken together, the two studies indicate that by 3 years of age, children do not indiscriminately imitate actions on a tool, but selectively reproduce those actions that have a desired causal effect.
---
paper_title: Causal knowledge and imitation/emulation switching in chimpanzees (Pan troglodytes) and children (Homo sapiens)

paper_content:

This study explored whether the tendency of chimpanzees and children to use emulation or imitation to solve a tool-using task was a response to the availability of causal information. Young wild-born chimpanzees from an African sanctuary and 3- to 4-year-old children observed a human demonstrator use a tool to retrieve a reward from a puzzle-box. The demonstration involved both causally relevant and irrelevant actions, and the box was presented in each of two conditions: opaque and clear. In the opaque condition, causal information about the effect of the tool inside the box was not available, and hence it was impossible to differentiate between the relevant and irrelevant parts of the demonstration. However, in the clear condition causal information was available, and subjects could potentially determine which actions were necessary. When chimpanzees were presented with the opaque box, they reproduced both the relevant and irrelevant actions, thus imitating the overall structure of the task. When the box was presented in the clear condition they instead ignored the irrelevant actions in favour of a more efficient, emulative technique. These results suggest that emulation is the favoured strategy of chimpanzees when sufficient causal information is available. However, if such information is not available, chimpanzees are prone to employ a more comprehensive copy of an observed action. In contrast to the chimpanzees, children employed imitation to solve the task in both conditions, at the expense of efficiency. We suggest that the difference in performance of chimpanzees and children may be due to a greater susceptibility of children to cultural conventions, perhaps combined with a differential focus on the results, actions and goals of the demonstrator
---
paper_title: Precision of imitation as a function of preschoolers' understanding of the goal of the demonstration.

paper_content:

The authors argue that imitation is a flexible and adaptive learning mechanism in that children do not always reproduce all of the details they can from a demonstration. Instead, they vary their replications depending on their interpretation of the situation. Specifically, the authors propose that when children do not understand the overall reason for a model's behavior, they will be more likely to imitate precisely. By copying conservatively in these situations, children may have a good chance of reproducing the action of the model correctly. In contrast, when the reason for an action is clear, children will be more likely to deviate from the manners and flourishes of the model and use their own means to complete the action. Imitation is an early developing, complex ability that children can use for a variety of purposes. We argue that imitation is a flexible and adaptive learning mechanism that children use differ- ently depending on the situation. In particular, our proposal is that children will imitate differently depending on how well they understand the demonstration at hand. When the reason for a demonstration is clear, we hypothesize that children will deviate from a model's precise manners in favor of their own means. When the overall goal of an action is not obvious, however, children should imitate conservatively, retaining more of the exact actions used by the model. In doing so, children will be more likely to successfully complete the intended action, even with a limited understanding of its purpose. Imitation, in general, has long been recognized as a rapid and efficient means to acquire information. Bandura (1986), for exam- ple, noted that by using observation, a learner can benefit from the experiences of others and avoid the painstaking and sometimes dangerous process of trial-and-error learning. Recently much the- oretical and experimental attention has focused more specifically on the importance of the ability to imitate precise actions—that is, the movements and means used to achieve a goal. In a study by Meltzoff (1988b), for example, 14-month-olds saw a model bend at the waist and turn on a light with the head. When presented with the light, approximately two thirds of the children also used this unusual manner to turn the light on even though more common
---
paper_title: Copying actions and copying outcomes: Social learning through the second year.

paper_content:

The present work documents how the logic of a model's demonstration and the communicative cues that the model provides interact with age to influence how children engage in social learning. Children at ages 12, 18, and 24 months (n = 204) watched a model open a series of boxes. Twelve-month-old subjects only copied the specific actions of the model when they were given a logical reason to do so- otherwise, they focused on reproducing the outcome of the demonstrated actions. Eighteen-month-old subjects focused on copying the outcome when the model was aloof. When the model acted socially, the subjects were as likely to focus on copying actions as outcomes, irrespective of the apparent logic of the model's behavior. Finally, 24-month-old subjects predominantly focused on copying the model's specific actions. However, they were less likely to produce the modeled outcome when the model acted nonsocially.
---
paper_title: Push or Pull: Imitation vs. Emulation in Great Apes and Human Children

paper_content:

All four species of great apes and young human children (12–24 mo of age) were administered an imitation task designed to distinguish between results learning (emulation) and action learning (imitation). Some subjects were exposed to a demonstrator either pushing or pulling a door to open a box, whereas others simply saw the door of the box opening itself in one of the two directions (the ghost control). Most of the apes successfully opened the box in both experimental conditions, as well as in a baseline condition, but without being influenced either by the demonstrator’s actions or by the door’s motions. In contrast, human children over 12 mo of age were influenced by the demonstration: the 18-mo-olds were influenced by the demonstrator’s actions, and the 24-mo-olds were influenced both by the demonstrator’s actions and by the door’s motions in the ghost control. These results provide support for the hypothesis that human children have a greater propensity than great apes for focusing either on a demonstrator’s action or on the result of their action, as needed, in social learning situations.
---
paper_title: Imitation of causally opaque versus causally transparent tool use by 3- and 5-year-old children

paper_content:

We investigated whether the tendency to imitate or emulate is influenced by the availability of causal information, and the amount of information available in a display. Three and 5-year-old children were shown by either a live or video model how to obtain a reward from either a clear or an opaque puzzle box. Some of the actions in the sequence were causally relevant to retrieving the reward, whereas others were irrelevant. The clear box made the causally irrelevant actions visible, whereas the opaque box prevented them from being seen. Results indicated that both 3- and 5-year-old children imitated the irrelevant actions regardless of the availability of causal information following a live demonstration. In contrast, the 3-year-olds employed an emulative approach, omitting irrelevant actions, when the information available was degraded in a video demonstration. However, the 5-year-olds were unaffected by the degraded information and employed an imitative approach. We suggest that imitation develops to be such an adaptive human strategy that it may often be employed at the expense of task efficiency.
---
paper_title: Imitative learning of artificial fruit processing in children (Homo sapiens) and chimpanzees (Pan troglodytes).

paper_content:

Observational learning in chimpanzees and young children was investigated using an artificial fruit designed as an analog of natural foraging problems faced by primates. Each of 3 principal components could be removed in 2 alternative ways, demonstration of only one of which was watched by each subject. This permitted subsequent imitation by subjects to be distinguished from stimulus enhancement. Children aged 2–4 years evidenced imitation for 2 components, but also achieved demonstrated outcomes through their own techniques. Chimpanzees relied even more on their own techniques, but they did imitate elements of 1 component of the task. To our knowledge, this is the first experimental evidence of chimpanzee imitation in a functional task designed to simulate foraging behavior hypothesized to be transmitted culturally in the wild.
---
paper_title: How do apes ape?

paper_content:

In the wake of telling critiques of the foundations on which earlier conclusions were based, the last 15 years have witnessed a renaissance in the study of social learning in apes. As a result, we are able to review 31 experimental studies from this period in which social learning in chimpanzees, gorillas, and orangutans has been investigated. The principal question framed at the beginning of this era, Do apes ape? has been answered in the affirmative, at least in certain conditions. The more interesting question now is, thus, How do apes ape? Answering this question has engendered richer taxonomies of the range of social-learning processes at work and new methodologies to uncover them. Together, these studies suggest that apes ape by employing a portfolio of alternative social-learning processes in flexibly adaptive ways, in conjunction with nonsocial learning. We conclude by sketching the kind of decision tree that appears to underlie the deployment of these alternatives.
---
paper_title: Doing the Right Thing: Infants' Selection of Actions to Imitate From Observed Event Sequences

paper_content:

Two studies were conducted to investigate how 14- to 16-month-old infants select actions to imitate from the stream of events. In each study, an experimenter demonstrated two actions leading to an interesting effect. Aspects of the first action were manipulated and whether infants performed this action when given the objects was observed. In both studies, infants were more likely to imitate the first action when it was physically necessary to generate the effect, and in Study 2 they were also more likely to imitate the action when it was socially cued. It seems that infants' own knowledge of space and causality as well as their sensitivity to others' social signals both contribute to their tendency to imitate actions.
---
paper_title: Causal knowledge and imitation/emulation switching in chimpanzees (Pan troglodytes) and children (Homo sapiens)

paper_content:

This study explored whether the tendency of chimpanzees and children to use emulation or imitation to solve a tool-using task was a response to the availability of causal information. Young wild-born chimpanzees from an African sanctuary and 3- to 4-year-old children observed a human demonstrator use a tool to retrieve a reward from a puzzle-box. The demonstration involved both causally relevant and irrelevant actions, and the box was presented in each of two conditions: opaque and clear. In the opaque condition, causal information about the effect of the tool inside the box was not available, and hence it was impossible to differentiate between the relevant and irrelevant parts of the demonstration. However, in the clear condition causal information was available, and subjects could potentially determine which actions were necessary. When chimpanzees were presented with the opaque box, they reproduced both the relevant and irrelevant actions, thus imitating the overall structure of the task. When the box was presented in the clear condition they instead ignored the irrelevant actions in favour of a more efficient, emulative technique. These results suggest that emulation is the favoured strategy of chimpanzees when sufficient causal information is available. However, if such information is not available, chimpanzees are prone to employ a more comprehensive copy of an observed action. In contrast to the chimpanzees, children employed imitation to solve the task in both conditions, at the expense of efficiency. We suggest that the difference in performance of chimpanzees and children may be due to a greater susceptibility of children to cultural conventions, perhaps combined with a differential focus on the results, actions and goals of the demonstrator
---
paper_title: The hidden structure of overimitation

paper_content:

Abstract ::: Young children are surprisingly judicious imitators, but there are also times when their reproduction of others' actions appears strikingly illogical. For example, children who observe an adult inefficiently operating a novel object frequently engage in what we term overimitation, persistently reproducing the adult's unnecessary actions. Although children readily overimitate irrelevant actions that even chimpanzees ignore, this curious effect has previously attracted little interest; it has been assumed that children overimitate not for theoretically significant reasons, but rather as a purely social exercise. In this paper, however, we challenge this view, presenting evidence that overimitation reflects a more fundamental cognitive process. We show that children who observe an adult intentionally manipulating a novel object have a strong tendency to encode all of the adult's actions as causally meaningful, implicitly revising their causal understanding of the object accordingly. This automatic causal encoding process allows children to rapidly calibrate their causal beliefs about even the most opaque physical systems, but it also carries a cost. When some of the adult's purposeful actions are unnecessary—even transparently so—children are highly prone to mis-encoding them as causally significant. The resulting distortions in children's causal beliefs are the true cause of overimitation, a fact that makes the effect remarkably resistant to extinction. Despite countervailing task demands, time pressure, and even direct warnings, children are frequently unable to avoid reproducing the adult's irrelevant actions because they have already incorporated them into their representation of the target object's causal structure. ::: ::: causal learning ::: cognitive development ::: imitation
---
paper_title: Infant imitation after a 1-week delay: Long-term memory for novel acts and multiple stimuli

paper_content:

Deferred imitation after a l-week delay was examined in 14-month-old infants. Six actions, each using a different object, were demonstrated to each infant. One of the six actions was a novel behavior that had a zero probability of occurrence in spontaneous play. In the imitation condition, infants observed the demonstration but were not allowed to touch the objects, thus preventing any immediate imitation. After the 1-week delay, infants returned to the laboratory and their imitation of the adult's previous actions was scored. Infants in the imitation condition produced significantly more of the target actions than infants in control groups who were not exposed to the modeling; there was also strong evidence for the imitation of the novel act. From a cognitive perspective deferred imitation provides a means of assessing recall memory and representation in children. From a socialdevelopmental viewpoint the findings illustrate that the behavioral repertoire of infants and their knowledge about objects can expand as a result of seeing the actions of others. Developmental theories differ in the extent to which imitation is viewed as making a contribution to the growth of a child's behavioral repertoire. Piaget (1952, 1954) tended to emphasize individual discovery and to minimize social imitation as a mechanism for acquiring new behaviors. Conversely, Bandura (1969, 1986) highlighted imitation as a channel for learning in school-age children, and Meltzoff (1985, 1988a, 1988b, in press) has emphasized the powerful influence of imitation in infancy. Regardless of the differing perspectives on the function of imitation, there is broad agreement that infants' ability to perform imitation after a delay, and especially deferred imitation of novel behaviors, is a milestone achievement both from a cognitive and a social perspective (Bandura, 1986; Flavell, 1985; Meltzoff, 1988b; Piaget, 1962). The acquisition of new behaviors through observation, particularly when they can delay production until significantly later, potentially provides both an avenue for social learning and a nonverbal measure of memory and representation. Although imitation after a delay has been actively investigated in school-age children (Bandura, Grusec, & Menlove, 1966; Bandura & Waiters, 1963), there have been only a few such experimental investigations in infants. McCall, Parke, and Kavanaugh (1977) reported imitation after a 1-day delay in 26month-old subjects. Meltzoff (1985) found deferred imitation of a simple action after a 1-day delay in 14-month-olds. Recently, it has been reported that infants as young as 9 months
---
paper_title: Causal knowledge and imitation/emulation switching in chimpanzees (Pan troglodytes) and children (Homo sapiens)

paper_content:

This study explored whether the tendency of chimpanzees and children to use emulation or imitation to solve a tool-using task was a response to the availability of causal information. Young wild-born chimpanzees from an African sanctuary and 3- to 4-year-old children observed a human demonstrator use a tool to retrieve a reward from a puzzle-box. The demonstration involved both causally relevant and irrelevant actions, and the box was presented in each of two conditions: opaque and clear. In the opaque condition, causal information about the effect of the tool inside the box was not available, and hence it was impossible to differentiate between the relevant and irrelevant parts of the demonstration. However, in the clear condition causal information was available, and subjects could potentially determine which actions were necessary. When chimpanzees were presented with the opaque box, they reproduced both the relevant and irrelevant actions, thus imitating the overall structure of the task. When the box was presented in the clear condition they instead ignored the irrelevant actions in favour of a more efficient, emulative technique. These results suggest that emulation is the favoured strategy of chimpanzees when sufficient causal information is available. However, if such information is not available, chimpanzees are prone to employ a more comprehensive copy of an observed action. In contrast to the chimpanzees, children employed imitation to solve the task in both conditions, at the expense of efficiency. We suggest that the difference in performance of chimpanzees and children may be due to a greater susceptibility of children to cultural conventions, perhaps combined with a differential focus on the results, actions and goals of the demonstrator
---
paper_title: Twelve- and 18-month-olds copy actions in terms of goals

paper_content:

In the context of an imitation game, 12- and 18-month-old infants saw an adult do such things as make a toy mouse hop across a mat (with sound effects). In one condition (House), the adult ended by placing the mouse in a toy house, whereas in another condition (No House) there was no house present at the final location. Infants at both ages usually simply put the mouse in the house (ignoring the hopping motion and sound effects) in the House condition, presumably because they interpreted the adult's action in terms of this final goal and so ignored the behavioral means. In contrast, infants copied the adult's action (both the hopping motion and the sound effects) when no house was present, presumably because here infants saw the action itself as the adult's only goal. From very early, infants’ social learning is flexible: infants focus on and copy either the end or the means of an adult action as required by the context.
---
paper_title: 'obsessed with goals' : Functions and mechanisms of teleological interpretation of actions in humans

paper_content:

Humans show a strong and early inclination to interpret observed behaviours of others as goal-directed actions. We identify two main epistemic functions that this ‘teleological obsession’ serves: on-line prediction and social learning. We show how teleological action interpretations can serve these functions by drawing on two kinds of inference (‘action-to-goal’ or ‘goal-to-action’), and argue that both types of teleological inference constitute inverse problems that can only be solved by further assumptions. We pinpoint the assumptions that the three currently proposed mechanisms of goal attribution (action-effect associations, simulation procedures, and teleological reasoning) imply, and contrast them with the functions they are supposed to fulfil. We argue that while action-effect associations and simulation procedures are generally well suited to serve on-line action monitoring and prediction, social learning of new means actions and artefact functions requires the inferential productivity of teleological reasoning.
---
paper_title: Visual transformations in gesture imitation: what you see is what you do

paper_content:

We propose an approach for a robot to imitate the gestures of a human demonstrator. Our framework consists solely of two components: a Sensory-Motor Map (SMM) and a View-Point Transformation (VPT). The SMM establishes an association between an arm image and the corresponding joint angles and it is learned by the system during a period of observation of its own gestures. The VPT is widely discussed in the psychology of visual perception and is used to transform the image of the demonstrator's arm to the so-called ego-centric image, as if the robot were observing its own arm. Different structures of the SMM and VPT are proposed in accordance with observations in human imitation. The whole system relies on monocular visual information and leads to a parsimonious architecture for learning by imitation. Real-time results are presented and discussed.
---
paper_title: A unifying computational framework for motor control and social interaction

paper_content:

Recent empirical studies have implicated the use of the motor system during action observation, imitation and social interaction. In this paper, we explore the computational parallels between the processes that occur in motor control and in action observation, imitation, social interaction and theory of mind. In particular, we examine the extent to which motor commands acting on the body can be equated with communicative signals acting on other people and suggest that computational solutions for motor control may have been extended to the domain of social interaction.
---
paper_title: A.: View sensitive cells as a neural basis for the representation of others in a self-centered frame of reference

paper_content:

This work follows from a research project, in which we investigate the underlying mechanisms of human imitation and develop a neural model of its core neural circuits. The present paper presents a model of a neural mechanism by which an imitator agent can map movements of the end effector performed by other agents onto its own frame of reference. The model mechanism is validated in simulation and in a humanoid robot to perform a simple task, in which the robot imitates movements performed a human demonstrator.
---
paper_title: Is imitation learning the route to humanoid robots?

paper_content:

Summary This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It will be postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in form of movement primitives. It will be reviewed how research on representations of, and functional connections between action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning will also be described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical learning research. Parallels and differences between biological and computational approaches to imitation will be highlighted and an overview of current projects that actually employ imitation learning for humanoid robots will be given.
---
paper_title: A Developmental Roadmap for Learning by Imitation in Robots

paper_content:

In this paper, we present a strategy whereby a robot acquires the capability to learn by imitation following a developmental pathway consisting on three levels: 1) sensory-motor coordination; 2) world interaction; and 3) imitation. With these stages, the system is able to learn tasks by imitating human demonstrators. We describe results of the different developmental stages, involving perceptual and motor skills, implemented in our humanoid robot, Baltazar. At each stage, the system's attention is drawn toward different entities: its own body and, later on, objects and people. Our main contributions are the general architecture and the implementation of all the necessary modules until imitation capabilities are eventually acquired by the robot. Also, several other contributions are made at each level: learning of sensory-motor maps for redundant robots, a novel method for learning how to grasp objects, and a framework for learning task description from observation for program-level imitation. Finally, vision is used extensively as the sole sensing modality (sometimes in a simplified setting) avoiding the need for special data-acquisition hardware
---
paper_title: Action, State and Effect Metrics for Robot Imitation

paper_content:

This paper addresses the problem of body mapping in robotic imitation where the demonstrator and imitator may not share the same embodiment (degrees of freedom (DOFs), body morphology, constraints, affordances and so on). Body mappings are formalized using a unified (linear) approach via correspondence matrices, which allow one to capture partial, mirror symmetric, one-to-one, one-to-many, many-to-one and many-to-many associations between various DOFs across dissimilar embodiments. We show how metrics for matching state and action aspects of behaviour can be mathematically determined by such correspondence mappings, which may serve to guide a robotic imitator. The approach is illustrated in a number of examples, using agents described by simple kinematic models and different types of correspondence mappings. Also, focusing on aspects of displacement and orientation of manipulated objects, a selection of metrics are presented, towards a characterization of the space of effect metrics.
---
paper_title: Generating whole body motions for a biped humanoid robot from captured human dances

paper_content:

The goal of this study is a system for a robot to imitate human dances. This paper describes the process to generate whole body motions which can be performed by an actual biped humanoid robot. Human dance motions are acquired through a motion capturing system. We then extract symbolic representation which is made up of primitive motions: essential postures in arm motions and step primitives in leg motions. A joint angle sequence of the robot is generated according to these primitive motions. Then joint angles are modified to satisfy mechanical constraints of the robot. For balance control, the waist trajectory is moved to acquire dynamics consistency based on desired ZMP. The generated motion is tested on OpenHRP dynamics simulator. In our test, the Japanese folk dance, 'Jongara-bushi', was successfully performed by HRP-1S.
---
paper_title: Visual learning by imitation with motor representations

paper_content:

We propose a general architecture for action (mimicking) and program (gesture) level visual imitation. Action-level imitation involves two modules. The viewpoint transformation (VPT) performs a "rotation" to align the demonstrator's body to that of the learner. The visuo-motor map (VMM) maps this visual information to motor data. For program-level (gesture) imitation, there is an additional module that allows the system to recognize and generate its own interpretation of observed gestures to produce similar gestures/goals at a later stage. Besides the holistic approach to the problem, our approach differs from traditional work in i) the use of motor information for gesture recognition; ii) usage of context (e.g., object affordances) to focus the attention of the recognition system and reduce ambiguities, and iii) use iconic image representations for the hand, as opposed to fitting kinematic models to the video sequence. This approach is motivated by the finding of visuomotor neurons in the F5 area of the macaque brain that suggest that gesture recognition/imitation is performed in motor terms (mirror) and rely on the use of object affordances (canonical) to handle ambiguous actions. Our results show that this approach can outperform more conventional (e.g., pure visual) methods.
---
paper_title: Visual transformations in gesture imitation: what you see is what you do

paper_content:

We propose an approach for a robot to imitate the gestures of a human demonstrator. Our framework consists solely of two components: a Sensory-Motor Map (SMM) and a View-Point Transformation (VPT). The SMM establishes an association between an arm image and the corresponding joint angles and it is learned by the system during a period of observation of its own gestures. The VPT is widely discussed in the psychology of visual perception and is used to transform the image of the demonstrator's arm to the so-called ego-centric image, as if the robot were observing its own arm. Different structures of the SMM and VPT are proposed in accordance with observations in human imitation. The whole system relies on monocular visual information and leads to a parsimonious architecture for learning by imitation. Real-time results are presented and discussed.
---
paper_title: Affordance-based imitation learning in robots

paper_content:

In this paper we build an imitation learning algorithm for a humanoid robot on top of a general world model provided by learned object affordances. We consider that the robot has previously learned a task independent affordance-based model of its interaction with the world. This model is used to recognize the demonstration by another agent (a human) and infer the task to be learned. We discuss several important problems that arise in this combined framework, such as the influence of an inaccurate model in the recognition of the demonstration. We illustrate the ideas in the paper with some experimental results obtained with a real robot.
---
paper_title: Reinforcement learning with limited reinforcement: using Bayes risk for active learning in POMDPs

paper_content:

Partially Observable Markov Decision Processes (POMDPs) have succeeded in planning domains that require balancing actions that increase an agent's knowledge and actions that increase an agent's reward. Unfortunately, most POMDPs are defined with a large number of parameters which are difficult to specify only from domain knowledge. In this paper, we present an approximation approach that allows us to treat the POMDP model parameters as additional hidden state in a "model-uncertainty" POMDP. Coupled with model-directed queries, our planner actively learns good policies. We demonstrate our approach on several POMDP problems.
---
paper_title: Bayesian Inverse Reinforcement Learning

paper_content:

Inverse Reinforcement Learning (IRL) is the problem of learning the reward function underlying a Markov Decision Process given the dynamics of the system and the behaviour of an expert. IRL is motivated by situations where knowledge of the rewards is a goal by itself (as in preference elicitation) and by the task of apprenticeship learning (learning policies from an expert). In this paper we show how to combine prior knowledge and evidence from the expert's actions to derive a probability distribution over the space of reward functions. We present efficient algorithms that find solutions for the reward learning and apprenticeship learning tasks that generalize well over these distributions. Experimental results show strong improvement for our methods over previous heuristic-based approaches.
---
paper_title: Robotic imitation from human motion capture using gaussian processes

paper_content:

Programming by demonstration, also called "imitation learning," offers the possibility of flexible, easily modifiable robotic systems. Full-fledged robotic imitation learning comprises many difficult subtasks. However, we argue that, at its core, imitation learning reduces to a regression problem. We propose a two-step framework in which an imitating agent first performs a regression from a high-dimensional observation space to a low-dimensional latent variable space. In the second step, the agent performs a regression from the latent variable space to a high-dimensional space representing degrees of freedom of its motor system. We demonstrate the validity of the approach by learning to map motion capture data from human actors to a humanoid robot. We also contrast use of several low-dimensional latent variable spaces, each covering a subset of agents' degrees of freedom, with use of a single, higher-dimensional latent variable space. Our findings suggest that compositing several regression models together yields qualitatively better imitation results than using a single, more complex regression model
---
paper_title: Affordances, development and imitation

paper_content:

We present a developmental perspective of robot learning that uses affordances as the link between sensory-motor coordination and imitation. The key concept is a general model for affordances able to learn the statistical relations between actions, object properties and the effects of actions on objects. Based on the learned affordances, it is possible to perform simple imitation games providing both task interpretation and planning capabilities. To evaluate the approach, we provide results of affordance learning with a real robot and simple imitation games with people.
---
paper_title: Visual transformations in gesture imitation: what you see is what you do

paper_content:

We propose an approach for a robot to imitate the gestures of a human demonstrator. Our framework consists solely of two components: a Sensory-Motor Map (SMM) and a View-Point Transformation (VPT). The SMM establishes an association between an arm image and the corresponding joint angles and it is learned by the system during a period of observation of its own gestures. The VPT is widely discussed in the psychology of visual perception and is used to transform the image of the demonstrator's arm to the so-called ego-centric image, as if the robot were observing its own arm. Different structures of the SMM and VPT are proposed in accordance with observations in human imitation. The whole system relies on monocular visual information and leads to a parsimonious architecture for learning by imitation. Real-time results are presented and discussed.
---
paper_title: A unifying computational framework for motor control and social interaction

paper_content:

Recent empirical studies have implicated the use of the motor system during action observation, imitation and social interaction. In this paper, we explore the computational parallels between the processes that occur in motor control and in action observation, imitation, social interaction and theory of mind. In particular, we examine the extent to which motor commands acting on the body can be equated with communicative signals acting on other people and suggest that computational solutions for motor control may have been extended to the domain of social interaction.
---
paper_title: Action recognition in the premotor cortex.

paper_content:

We recorded electrical activity from 532 neurons in the rostral part of inferior area 6 (area F5) of two macaque monkeys. Previous data had shown that neurons of this area discharge during goal-directed hand and mouth movements. We describe here the properties of a newly discovered set of F5 neurons (‘mirror neurons’, n = 92) all of which became active both when the monkey performed a given action and when it observed a similar action performed by the experimenter. Mirror neurons, in order to be visually triggered, required an interaction between the agent of the action and the object of it. The sight of the agent alone or of the object alone (three-dimensional objects, food) were ineffective. Hand and the mouth were by far the most effective agents. The actions most represented among those activating mirror neurons were grasping, manipulating and placing. In most mirror neurons (92%) there was a clear relation between the visual action they responded to and the motor response they coded. In ∼30% of mirror neurons the congruence was very strict and the effective observed and executed actions corresponded both in terms of general action (e.g. grasping) and in terms of the way in which that action was executed (e.g. precision grip). We conclude by proposing that mirror neurons form a system for matching observation and execution of motor actions. We discuss the possible role of this system in action recognition and, given the proposed homology between F5 and human Brocca’s region, we posit that a matching system, similar to that of mirror neurons exists in humans and could be involved in recognition of actions as well as phonetic gestures.
---
paper_title: Active Imitation Learning

paper_content:

Imitation learning, also called learning by watching or programming by demonstration, has emerged as a means of accelerating many reinforcement learning tasks. Previous work has shown the value of imitation in domains where a single mentor demonstrates execution of a known optimal policy for the benefit of a learning agent. We consider the more general scenario of learning from mentors who are themselves agents seeking to maximize their own rewards. We propose a new algorithm based on the concept of transferable utility for ensuring that an observer agent can learn efficiently in the context of a selfish, not necessarily helpful, mentor. We also address the questions of when an imitative agent should request help from a mentor, and when the mentor can be expected to acknowledge a request for help. In analogy with other types of active learning, we call the proposed approach active imitation learning.
---
paper_title: Learning Object Affordances: From Sensory--Motor Coordination to Imitation

paper_content:

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.
---
paper_title: Causal knowledge and imitation/emulation switching in chimpanzees (Pan troglodytes) and children (Homo sapiens)

paper_content:

This study explored whether the tendency of chimpanzees and children to use emulation or imitation to solve a tool-using task was a response to the availability of causal information. Young wild-born chimpanzees from an African sanctuary and 3- to 4-year-old children observed a human demonstrator use a tool to retrieve a reward from a puzzle-box. The demonstration involved both causally relevant and irrelevant actions, and the box was presented in each of two conditions: opaque and clear. In the opaque condition, causal information about the effect of the tool inside the box was not available, and hence it was impossible to differentiate between the relevant and irrelevant parts of the demonstration. However, in the clear condition causal information was available, and subjects could potentially determine which actions were necessary. When chimpanzees were presented with the opaque box, they reproduced both the relevant and irrelevant actions, thus imitating the overall structure of the task. When the box was presented in the clear condition they instead ignored the irrelevant actions in favour of a more efficient, emulative technique. These results suggest that emulation is the favoured strategy of chimpanzees when sufficient causal information is available. However, if such information is not available, chimpanzees are prone to employ a more comprehensive copy of an observed action. In contrast to the chimpanzees, children employed imitation to solve the task in both conditions, at the expense of efficiency. We suggest that the difference in performance of chimpanzees and children may be due to a greater susceptibility of children to cultural conventions, perhaps combined with a differential focus on the results, actions and goals of the demonstrator
---
paper_title: Generating whole body motions for a biped humanoid robot from captured human dances

paper_content:

The goal of this study is a system for a robot to imitate human dances. This paper describes the process to generate whole body motions which can be performed by an actual biped humanoid robot. Human dance motions are acquired through a motion capturing system. We then extract symbolic representation which is made up of primitive motions: essential postures in arm motions and step primitives in leg motions. A joint angle sequence of the robot is generated according to these primitive motions. Then joint angles are modified to satisfy mechanical constraints of the robot. For balance control, the waist trajectory is moved to acquire dynamics consistency based on desired ZMP. The generated motion is tested on OpenHRP dynamics simulator. In our test, the Japanese folk dance, 'Jongara-bushi', was successfully performed by HRP-1S.
---
paper_title: Visual learning by imitation with motor representations

paper_content:

We propose a general architecture for action (mimicking) and program (gesture) level visual imitation. Action-level imitation involves two modules. The viewpoint transformation (VPT) performs a "rotation" to align the demonstrator's body to that of the learner. The visuo-motor map (VMM) maps this visual information to motor data. For program-level (gesture) imitation, there is an additional module that allows the system to recognize and generate its own interpretation of observed gestures to produce similar gestures/goals at a later stage. Besides the holistic approach to the problem, our approach differs from traditional work in i) the use of motor information for gesture recognition; ii) usage of context (e.g., object affordances) to focus the attention of the recognition system and reduce ambiguities, and iii) use iconic image representations for the hand, as opposed to fitting kinematic models to the video sequence. This approach is motivated by the finding of visuomotor neurons in the F5 area of the macaque brain that suggest that gesture recognition/imitation is performed in motor terms (mirror) and rely on the use of object affordances (canonical) to handle ambiguous actions. Our results show that this approach can outperform more conventional (e.g., pure visual) methods.
---
paper_title: A model for inferring the intention in imitation tasks

paper_content:

Robot imitation comprises a number of hard problems, one of the most difficult problems is the extraction of intention from a demonstration. A demonstration can almost always be interpreted in different ways, making it impossible for the imitating robot to find out what exactly it is that was demonstrated. We first attempt to set out the problem of intention reading. Next, we offer a computational model which implements a solution to intention reading. Our model needs repeated interactions between the demonstrator and the imitator. Through keeping a score about which interactions where successful, the imitating robot gradually builds a model which ?understands? what the intent is of the demonstrator.
---
paper_title: Discovering optimal imitation strategies

paper_content:

This paper develops a general policy for learning the relevant features of an imitation task. We restrict our study to imitation of manipulative tasks or gestures. The imitation process is modeled as a hierarchical optimization system, which minimizes the discrepancy between two multi-dimensional datasets. To classify across manipulation strategies, we apply a probabilistic analysis to data in Cartesian and joint spaces. We determine a general metric that optimizes the policy of task reproduction, following strategy determination. The model successfully discovers strategies in six different imitative tasks and controls task reproduction by a full body humanoid robot. © 2004 Elsevier B.V. All rights reserved.
---
paper_title: Imitation without intentionality. Using string parsing to copy the organization of behaviour

paper_content:

A theory of imitation is proposed, string parsing, which separates the copying of behavioural organization by observation from an understanding of the cause of its effectiveness. In string parsing, recurring patterns in the visible stream of behaviour are detected and used to build a statistical sketch of the underlying hierarchical structure. This statistical sketch may in turn aid the subsequent comprehension of cause and effect. Three cases of social learning of relatively complex skills are examined, as potential cases of imitation by string parsing. Understanding the basic requirements for successful string parsing helps to resolve the conflict between mainly negative reports of imitation in experiments and more positive evidence from natural conditions. Since string parsing does not depend on comprehension of the intentions of other agents or the everyday physics of objects, separate tests of these abilities are needed even in animals shown to learn by imitation.
---
paper_title: Investigating Action Understanding: Inferential Processes versus Action Simulation

paper_content:

In our daily life, we continuously monitor others' behaviors and interpret them in terms of goals, intentions, and reasons. Despite their central importance for predicting and interpreting each other's actions, the functional mechanisms and neural circuits involved in action understanding remain highly controversial. Two alternative accounts have been advanced. Simulation theory assumes that we understand actions by simulating the observed behavior through a direct matching process that activates the mirror-neuron circuit. The alternative interpretive account assumes that action understanding is based on specialized inferential processes activating brain areas with no mirror properties. Although both approaches recognize the central role of contextual information in specifying action intentions, their respective accounts of this process differ in significant respects. Here, we investigated the role of context in action understanding by using functional brain imaging while participants observed an unusual action in implausible versus plausible contexts. We show that brain areas that are part of a network involved in inferential interpretive processes of rationalization and mentalization but that lack mirror properties are more active when the action occurs in an implausible context. However, no differential activation was found in the mirror network. Our findings support the assumption that action understanding in novel situations is primarily mediated by an inferential interpretive system rather than the mirror system.
---
paper_title: A unifying computational framework for motor control and social interaction

paper_content:

Recent empirical studies have implicated the use of the motor system during action observation, imitation and social interaction. In this paper, we explore the computational parallels between the processes that occur in motor control and in action observation, imitation, social interaction and theory of mind. In particular, we examine the extent to which motor commands acting on the body can be equated with communicative signals acting on other people and suggest that computational solutions for motor control may have been extended to the domain of social interaction.
---
paper_title: Imitation with ALICE: Learning to Imitate Corresponding Actions across Dissimilar Embodiments

paper_content:

Imitation is a powerful mechanism whereby knowledge may be transferred between agents (both biological and artificial). Key problems on the topic of imitation have emerged in various areas close to artificial intelligence, including the cognitive and social sciences, animal behavior, robotics, human-computer interaction, embodied intelligence, software engineering, programming by example and machine learning. Artificial systems used to study imitation can both test models of imitation derived from observational or neurobiological data on imitation in animals and then apply them to different kinds of nonbiological systems ranging from robots to software agents. A crucial problem in imitation is the correspondence problem, mapping action sequences of the demonstrator and the imitator agent. This problem becomes particularly obvious when the two agents do not share the same embodiment and affordances. This paper describes a new general imitation mechanism called ALICE (action learning for imitation via correspondence between embodiments) that specifically addresses the correspondence problem. The mechanism is implemented and its efficacy illustrated on the "chessworld" testbed that was created to study imitation from an agent-based perspective, i.e., by a particular agent in a particular environment.
---
paper_title: Affordance-based imitation learning in robots

paper_content:

In this paper we build an imitation learning algorithm for a humanoid robot on top of a general world model provided by learned object affordances. We consider that the robot has previously learned a task independent affordance-based model of its interaction with the world. This model is used to recognize the demonstration by another agent (a human) and infer the task to be learned. We discuss several important problems that arise in this combined framework, such as the influence of an inaccurate model in the recognition of the demonstration. We illustrate the ideas in the paper with some experimental results obtained with a real robot.
---
paper_title: Is imitation learning the route to humanoid robots?

paper_content:

Summary This review investigates two recent developments in artificial intelligence and neural computation: learning from imitation and the development of humanoid robots. It will be postulated that the study of imitation learning offers a promising route to gain new insights into mechanisms of perceptual motor control that could ultimately lead to the creation of autonomous humanoid robots. Imitation learning focuses on three important issues: efficient motor learning, the connection between action and perception, and modular motor control in form of movement primitives. It will be reviewed how research on representations of, and functional connections between action and perception have contributed to our understanding of motor acts of other beings. The recent discovery that some areas in the primate brain are active during both movement perception and execution has provided a hypothetical neural basis of imitation. Computational approaches to imitation learning will also be described, initially from the perspective of traditional AI and robotics, but also from the perspective of neural network models and statistical learning research. Parallels and differences between biological and computational approaches to imitation will be highlighted and an overview of current projects that actually employ imitation learning for humanoid robots will be given.
---
paper_title: Evaluation Metrics and Results of Human Arm Movement Imitation

paper_content:

We present a psychophysical study of human arm movement imitation, and an approach to analyzing the resulting data, which is general enough to be applied to human or humanoid movement analysis. We describe a joint-space based segmentation and comparison algorithm that allows us to evaluate the performance of 11 different subjects performing a series of arm movement imitation tasks. The results provide analytical evidence for the strong interference effects of simultaneous rehearsal during observation. Additionally, the results also demonstrate that repeated imitation in these tasks did not affect the subjects' performance.
---
paper_title: Bayesian models of human action understanding

paper_content:

We present a Bayesian framework for explaining how people reason about and predict the actions of an intentional agent, based on observing its behavior. Action-understanding is cast as a problem of inverting a probabilistic generative model, which assumes that agents tend to act rationally in order to achieve their goals given the constraints of their environment. Working in a simple sprite-world domain, we show how this model can be used to infer the goal of an agent and predict how the agent will act in novel situations or when environmental constraints change. The model provides a qualitative account of several kinds of inferences that preverbal infants have been shown to perform, and also fits quantitative predictions that adult observers make in a new experiment.
---
paper_title: ONLINE LEARNING OF THE BODY SCHEMA

paper_content:

We present an algorithm enabling a humanoid robot to visually learn its body schema, knowing only the number of degrees of freedom in each limb. By "body schema" we mean the joint positions and orientations and thus the kinematic function. The learning is performed by visually observing its end-effectors when moving them. With simulations involving a body schema of more than 20 degrees of freedom, results show that the system is scalable to a high number of degrees of freedom. Real robot experiments confirm the practicality of our approach. Our results illustrate how subjective space representation can develop as a result of sensorimotor contingencies.
---
paper_title: Visual transformations in gesture imitation: what you see is what you do

paper_content:

We propose an approach for a robot to imitate the gestures of a human demonstrator. Our framework consists solely of two components: a Sensory-Motor Map (SMM) and a View-Point Transformation (VPT). The SMM establishes an association between an arm image and the corresponding joint angles and it is learned by the system during a period of observation of its own gestures. The VPT is widely discussed in the psychology of visual perception and is used to transform the image of the demonstrator's arm to the so-called ego-centric image, as if the robot were observing its own arm. Different structures of the SMM and VPT are proposed in accordance with observations in human imitation. The whole system relies on monocular visual information and leads to a parsimonious architecture for learning by imitation. Real-time results are presented and discussed.
---
paper_title: Reinforcement learning of motor skills with policy gradients

paper_content:

Autonomous learning is one of the hallmarks of human and animal behavior, and understanding the principles of learning will be crucial in order to achieve true autonomy in advanced machines like humanoid robots. In this paper, we examine learning of complex motor skills with human-like limbs. While supervised learning can offer useful tools for bootstrapping behavior, e.g., by learning from demonstration, it is only reinforcement learning that offers a general approach to the final trial-and-error improvement that is needed by each individual acquiring a skill. Neither neurobiological nor machine learning studies have, so far, offered compelling results on how reinforcement learning can be scaled to the high-dimensional continuous state and action spaces of humans or humanoids. Here, we combine two recent research developments on learning motor control in order to achieve this scaling. First, we interpret the idea of modular motor control by means of motor primitives as a suitable way to generate parameterized control policies for reinforcement learning. Second, we combine motor primitives with the theory of stochastic policy gradient learning, which currently seems to be the only feasible framework for reinforcement learning for humanoids. We evaluate different policy gradient methods with a focus on their applicability to parameterized motor primitives. We compare these algorithms in the context of motor primitive learning, and show that our most modern algorithm, the Episodic Natural Actor-Critic outperforms previous algorithms by at least an order of magnitude. We demonstrate the efficiency of this reinforcement learning method in the application of learning to hit a baseball with an anthropomorphic robot arm.
---
paper_title: Gaussian Process Dynamical Models for Human Motion

paper_content:

We introduce Gaussian process dynamical models (GPDMs) for nonlinear time series analysis, with applications to learning models of human pose and motion from high-dimensional motion capture data. A GPDM is a latent variable model. It comprises a low-dimensional latent space with associated dynamics, as well as a map from the latent space to an observation space. We marginalize out the model parameters in closed form by using Gaussian process priors for both the dynamical and the observation mappings. This results in a nonparametric model for dynamical systems that accounts for uncertainty in the model. We demonstrate the approach and compare four learning algorithms on human motion capture data, in which each pose is 50-dimensional. Despite the use of small data sets, the GPDM learns an effective representation of the nonlinear dynamics in these spaces.
---
paper_title: Adaptive Body Scheme Models for Robust Robotic Manipulation

paper_content:

Truly autonomous systems require the ability to monitor and adapt their internal body scheme throughout their entire lifetime. In this paper, we present an approach allowing a robot to learn from scratch and maintain a generative model of its own physical body through self-observation with a single monocular camera. We represent the robot’s internal model as a compact Bayesian network, consisting of local models that describe the physical relationships between neighboring body parts. We introduce a flexible Bayesian framework that allows to simultaneously select the maximum-likely network structure and to learn the underlying conditional density functions. Changes in the robot’s physiology can be detected by identifying mismatches between model predictions and the self-perception. To quickly adapt the model to changed situations, we developed an efficient search heuristic that starts from the structure of the best explaining memorized network and then replaces local components where necessary. In experiments carried out with a real robot equipped with a 6-DOF manipulator as well as in simulation, we show that our system can quickly adapt to changes of the body physiology in full 3D space, in particular with limited visibility, noisy and partially missing observations, and without the need for proprioception.
---
paper_title: Learning about objects through action - initial steps towards artificial cognition

paper_content:

Within the field of Neuro Robotics we are driven primarily by the desire to understand how humans and animals live and grow and solve every day's problems. To this aim we adopted a "learn by doing" approach by building artificial systems, e.g. robots that not only look like human beings but also represent a model of some brain process. They should, ideally, behave and interact like human beings (being situated). The main emphasis in robotics has been on systems that act as a reaction to an external stimulus (e.g. tracking, reaching), rather than as a result of an internal drive to explore or "understand" the environment. We think it is now appropriate to try to move from acting, in the sense explained above, to "understanding". As a starting point we addressed the problem of learning about the effects and consequences of self-generated actions. How does the robot learn how to pull an object toward itself or to push it away? How does the robot learn that spherical objects roll while a cube only slides if pushed? Interacting with objects is important because it implicitly explores object representation, event understanding, and can provide definition of object-hood that could not be grasped with a mere passive observation of the world. Further, learning to understand what one's own body can do is an essential step toward learning by imitation. In this view two actions are similar not only if their kinematics and dynamics are similar but rather if the effects on the external world are the same. Along this line of research we discuss some recent experiments performed at the AI-Lab at MIT and at the LIRA-Lab at the University of Genova on COG and Babybot respectively. We show how the humanoid robots can learn how to poke and prod objects to obtain a consistently repeatable effect (e.g. sliding in a given direction), to help visual segmentation, and to interpret a poking action performed by a human manipulator.
---
paper_title: To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control

paper_content:

The concept of affordances was introduced by J. J. Gibson to eXplain how inherent "values" and "meanings" of things in the environment can be directly perceived and how this information can be linked to the action possibilities offered to the organism by the environment. Although introduced in psychology, the concept influenced studies in other fields ranging from human—computer interaction to autonomous robotics. In this article, we first introduce the concept of affordances as conceived by J. J. Gibson and review the use of the term in different fields, with particular emphasis on its use in autonomous robotics. Then, we summarize four of the major formalization proposals for the affordance term. We point out that there are three, not one, perspectives from which to view affordances and that much of the confusion regarding discussions on the concept has arisen from this. We propose a new formalism for affordances and discuss its implications for autonomous robot control. We report preliminary results obtained with robots and link them with these implications.
---
paper_title: Learning movement primitives

paper_content:

This paper discusses a comprehensive framework for modular motor control based on a recently developed theory of dynamic movement primitives (DMP). DMPs are a formulation of movement primitives with autonomous nonlinear differential equations, whose time evolution creates smooth kinematic control policies. Model-based control theory is used to convert the outputs of these policies into motor commands. By means of coupling terms, on-line modifications can be incorporated into the time evolution of the differential equations, thus providing a rather flexible and reactive framework for motor planning and execution. The linear parameterization of DMPs lends itself naturally to supervised learning from demonstration. Moreover, the temporal, scale, and translation invariance of the differential equations with respect to these parameters provides a useful means for movement recognition. A novel reinforcement learning technique based on natural stochastic policy gradients allows a general approach of improving DMPs by trial and error learning with respect to almost arbitrary optimization criteria. We demonstrate the different ingredients of the DMP approach in various examples, involving skill learning from demonstration on the humanoid robot DB, and learning biped walking from demonstration in simulation, including self-improvement of the movement patterns towards energy efficiency through resonance tuning.
---
paper_title: Learning Object Affordances: From Sensory--Motor Coordination to Imitation

paper_content:

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.
---
paper_title: From motor babbling to hierarchical learning by imitation: A robot developmental pathway

paper_content:

How does an individual use the knowledge ::: acquired through self exploration as a manipulable model through which to understand ::: others and benefit from their knowledge? ::: How can developmental and social learning be ::: combined for their mutual benefit? In this ::: paper we review a hierarchical architecture ::: (HAMMER) which allows a principled way ::: for combining knowledge through exploration ::: and knowledge from others, through the creation and use of multiple inverse and forward ::: models. We describe how Bayesian Belief Networks can be used to learn the association ::: between a robot’s motor commands and sensory consequences (forward models), and how ::: the inverse association can be used for imitation. Inverse models created through self ::: exploration, as well as those from observing ::: others can coexist and compete in a principled unified framework, that utilises the simulation theory of mind approach to mentally ::: rehearse and understand the actions of others.
---
paper_title: A Developmental Roadmap for Learning by Imitation in Robots

paper_content:

In this paper, we present a strategy whereby a robot acquires the capability to learn by imitation following a developmental pathway consisting on three levels: 1) sensory-motor coordination; 2) world interaction; and 3) imitation. With these stages, the system is able to learn tasks by imitating human demonstrators. We describe results of the different developmental stages, involving perceptual and motor skills, implemented in our humanoid robot, Baltazar. At each stage, the system's attention is drawn toward different entities: its own body and, later on, objects and people. Our main contributions are the general architecture and the implementation of all the necessary modules until imitation capabilities are eventually acquired by the robot. Also, several other contributions are made at each level: learning of sensory-motor maps for redundant robots, a novel method for learning how to grasp objects, and a framework for learning task description from observation for program-level imitation. Finally, vision is used extensively as the sole sensing modality (sometimes in a simplified setting) avoiding the need for special data-acquisition hardware
---
paper_title: Learning nonparametric models for probabilistic imitation

paper_content:

Learning by imitation represents an important mechanism for rapid acquisition of new behaviors in humans and robots. A critical requirement for learning by imitation is the ability to handle uncertainty arising from the observation process as well as the imitator's own dynamics and interactions with the environment. In this paper, we present a new probabilistic method for inferring imitative actions that takes into account both the observations of the teacher as well as the imitator's dynamics. Our key contribution is a nonparametric learning method which generalizes to systems with very different dynamics. Rather than relying on a known forward model of the dynamics, our approach learns a nonparametric forward model via exploration. Leveraging advances in approximate inference in graphical models, we show how the learned forward model can be directly used to plan an imitating sequence. We provide experimental results for two systems: a biomechanical model of the human arm and a 25-degrees-of-freedom humanoid robot. We demonstrate that the proposed method can be used to learn appropriate motor inputs to the model arm which imitates the desired movements. A second set of results demonstrates dynamically stable full-body imitation of a human teacher by the humanoid robot.
---
paper_title: Learning inverse kinematics

paper_content:

Real-time control of the end-effector of a humanoid robot in external coordinates requires computationally efficient solutions of the inverse kinematics problem. In this context, this paper investigates inverse kinematics learning for resolved motion rate control (RMRC) employing an optimization criterion to resolve kinematic redundancies. Our learning approach is based on the key observations that learning an inverse of a nonuniquely invertible function can be accomplished by augmenting the input representation to the inverse model and by using a spatially localized learning approach. We apply this strategy to inverse kinematics learning and demonstrate how a recently developed statistical learning algorithm, locally weighted projection regression, allows efficient learning of inverse kinematic mappings in an incremental fashion even when input spaces become rather high dimensional. Our results are illustrated with a 30-DOF humanoid robot.
---
paper_title: From visuo-motor self learning to early imitation-a neural architecture for humanoid learning

paper_content:

Behavior imitation ability will be a key technology for future human friendly robots. In order to understand the principles and mechanisms of imitation, we take a synthetic cognitive developmental approach, starting with minimum components and create a system that can learn to imitate others. We developed a visuo-motor neural learning system which consists of orientation selective visual movement representation, distributed arm movement representation, and a high-dimensional temporal sequence learning mechanism. The vision and the movement representations model the findings in primate brain, i.e. macaque area MT(or human area V5) and the primary motor area. The learning mechanism is inspired by the finding that there are excessive connections in neonate brain. As our robot explores the visuo-motor self movement patterns, it learns coherent patterns as high-dimensional trajectory attractors. After the learning, a human comes in front of the robot showing arm movements which are similar to the ones in self learning. Although the robot has never seen or programmed to interpret human arm movement, and the detail of visual stimuli are very different, the robot identifies some of the patterns as similar to those in self learning, and responded by generating the previously learned arm movement. In other words, the robot exhibits early imitation ability based on self exploratory learning.
---
paper_title: Visual learning by imitation with motor representations

paper_content:

We propose a general architecture for action (mimicking) and program (gesture) level visual imitation. Action-level imitation involves two modules. The viewpoint transformation (VPT) performs a "rotation" to align the demonstrator's body to that of the learner. The visuo-motor map (VMM) maps this visual information to motor data. For program-level (gesture) imitation, there is an additional module that allows the system to recognize and generate its own interpretation of observed gestures to produce similar gestures/goals at a later stage. Besides the holistic approach to the problem, our approach differs from traditional work in i) the use of motor information for gesture recognition; ii) usage of context (e.g., object affordances) to focus the attention of the recognition system and reduce ambiguities, and iii) use iconic image representations for the hand, as opposed to fitting kinematic models to the video sequence. This approach is motivated by the finding of visuomotor neurons in the F5 area of the macaque brain that suggest that gesture recognition/imitation is performed in motor terms (mirror) and rely on the use of object affordances (canonical) to handle ambiguous actions. Our results show that this approach can outperform more conventional (e.g., pure visual) methods.
---
paper_title: A unifying computational framework for motor control and social interaction

paper_content:

Recent empirical studies have implicated the use of the motor system during action observation, imitation and social interaction. In this paper, we explore the computational parallels between the processes that occur in motor control and in action observation, imitation, social interaction and theory of mind. In particular, we examine the extent to which motor commands acting on the body can be equated with communicative signals acting on other people and suggest that computational solutions for motor control may have been extended to the domain of social interaction.
---
paper_title: Visual learning by imitation with motor representations

paper_content:

We propose a general architecture for action (mimicking) and program (gesture) level visual imitation. Action-level imitation involves two modules. The viewpoint transformation (VPT) performs a "rotation" to align the demonstrator's body to that of the learner. The visuo-motor map (VMM) maps this visual information to motor data. For program-level (gesture) imitation, there is an additional module that allows the system to recognize and generate its own interpretation of observed gestures to produce similar gestures/goals at a later stage. Besides the holistic approach to the problem, our approach differs from traditional work in i) the use of motor information for gesture recognition; ii) usage of context (e.g., object affordances) to focus the attention of the recognition system and reduce ambiguities, and iii) use iconic image representations for the hand, as opposed to fitting kinematic models to the video sequence. This approach is motivated by the finding of visuomotor neurons in the F5 area of the macaque brain that suggest that gesture recognition/imitation is performed in motor terms (mirror) and rely on the use of object affordances (canonical) to handle ambiguous actions. Our results show that this approach can outperform more conventional (e.g., pure visual) methods.
---
paper_title: Robot Learning by Active Imitation

paper_content:

A key area of robotics research is concerned with developing social robots for assisting humans in everyday tasks. Many of the motion skills required by the robot to perform such tasks can be pre-programmed. However, it is normally agreed that a truly useful robotic companion should be equipped with some learning capabilities, in order to adapt to unknown environments, or, what is most difficult, learn to perform new tasks. Many learning algorithms have been proposed for robotics applications. However, these learning algorithms are often task specific, and only work if the learning task is predefined in a delicate representation, and a set of pre-collected training samples is available. Besides, the distributions of training and test samples have to be identical and the world model is totally or partially given (Tan et al., 2005). In a human world, these conditions are commonly impossible to achieve. Therefore, these learning algorithms involve a process of optimization in a large search space in order to find the best behaviour fitting the observed samples, as well as some prior knowledge. If the task becomes more complicated or multiple tasks are involved, the search process is often incapable of satisfying real-time responses. Learning by observation and imitation constitute two important mechanisms for learning behaviours socially in humans and other animal species, e.g. dolphins, chimpanzees and other apes (Dautenhahn & Nehaniv, 2002). Inspired by nature, and in order to speed up the learning process in complex motor systems, Stefan Schaal appealed for a pragmatic view of imitation (Schaal, 1999) as a tool to improve the learning process. Current work has demonstrated that learning by observation and imitation is a powerful tool to acquire new abilities, which encourages social interaction and cultural transfer. It permits robots to quickly learn new skills and tasks from natural human instructions and few demonstrations (Alissandrakis et al., 2002, Breazeal et al., 2005, Demiris & Hayes, 2002, Sauser & Billard, 2005). In robotics, the ability to imitate relies upon the robot having many perceptual, cognitive and motor capabilities. The impressive advance of research and development in robotics over the past few years has led to the development of this type of robots, e.g. Sarcos (Ijspeert et al., 2002) or Kenta (Inaba et al., 2003). However, even if a robot has the necessary skills to imitate the human movement, most published work focus on specific components of an imitation system (Lopes & Santos-Victor, 2005). The development of a complete imitation architecture is difficult. Some of the main challenges are: how to identify which features of an action are important; how to reproduce such action; and how to evaluate the performance of the imitation process (Breazeal & Scassellati, 2002).
---
paper_title: Visual transformations in gesture imitation: what you see is what you do

paper_content:

We propose an approach for a robot to imitate the gestures of a human demonstrator. Our framework consists solely of two components: a Sensory-Motor Map (SMM) and a View-Point Transformation (VPT). The SMM establishes an association between an arm image and the corresponding joint angles and it is learned by the system during a period of observation of its own gestures. The VPT is widely discussed in the psychology of visual perception and is used to transform the image of the demonstrator's arm to the so-called ego-centric image, as if the robot were observing its own arm. Different structures of the SMM and VPT are proposed in accordance with observations in human imitation. The whole system relies on monocular visual information and leads to a parsimonious architecture for learning by imitation. Real-time results are presented and discussed.
---
paper_title: A.: View sensitive cells as a neural basis for the representation of others in a self-centered frame of reference

paper_content:

This work follows from a research project, in which we investigate the underlying mechanisms of human imitation and develop a neural model of its core neural circuits. The present paper presents a model of a neural mechanism by which an imitator agent can map movements of the end effector performed by other agents onto its own frame of reference. The model mechanism is validated in simulation and in a humanoid robot to perform a simple task, in which the robot imitates movements performed a human demonstrator.
---
paper_title: Visual learning by imitation with motor representations

paper_content:

We propose a general architecture for action (mimicking) and program (gesture) level visual imitation. Action-level imitation involves two modules. The viewpoint transformation (VPT) performs a "rotation" to align the demonstrator's body to that of the learner. The visuo-motor map (VMM) maps this visual information to motor data. For program-level (gesture) imitation, there is an additional module that allows the system to recognize and generate its own interpretation of observed gestures to produce similar gestures/goals at a later stage. Besides the holistic approach to the problem, our approach differs from traditional work in i) the use of motor information for gesture recognition; ii) usage of context (e.g., object affordances) to focus the attention of the recognition system and reduce ambiguities, and iii) use iconic image representations for the hand, as opposed to fitting kinematic models to the video sequence. This approach is motivated by the finding of visuomotor neurons in the F5 area of the macaque brain that suggest that gesture recognition/imitation is performed in motor terms (mirror) and rely on the use of object affordances (canonical) to handle ambiguous actions. Our results show that this approach can outperform more conventional (e.g., pure visual) methods.
---
paper_title: Imitation learning based on visuo-somatic mapping

paper_content:

Imitation learning is a powerful approach to humanoid behavior generation, however, the most existing methods assume the availability of the information on the internal state of a demonstrator such as joint angles, while humans usually cannot directly access to imitate the observed behavior. This paper presents a method of imitation learning based on visuosomatic mapping from observing the demonstrator’s posture to reminding the self posture via mapping from the self motion observation to the self posture for both motion understanding and generation. First, various kinds of posture data of the observer are mapped onto posture space by self organizing mapping (hereafter, SOM), and the trajectories in the posture space are mapped onto a motion segment space by SOM again for data reduction. Second, optical flows caused by the demonstrator’s motions or the self motions are mapped onto a flow segment space where parameterized flow data are connected with the corresponding motion segments in the motion segment space. The connection with the self motion is straightforward, and is easily acquired by Hebbian Learning. Then, the connection with the demonstrator’s motion is automatic based on the learned connection. Finally, the visuo-somatic mapping is completed when the posture space (the observer: self) and image space (the demonstrator: other) are connected, which means observing the demonstrator’s posture associcates the self posture. Experimental results with human motion data are shown and the discussion is given with future issues.
---
paper_title: What should be taught first: the emotional expression or the face?

paper_content:

We are interested in knowing how a robot head can learn to recognize facial expressions without supervision. Our starting point is a mathematical model showing that a sensory-motor architecture is able to express its emotions succeedes to recognize on-line the facial expression of a caregiver if this latter naturally tends to imitate or to resonate with the system. Interestingly, our works also show that, learning autonomously to recognize a face/non face is more complex than to recognize a facial expression. We propose an architecture using the interaction rhythm to allow first a robust learning of the facial expression without a face tracking and next to perform the learning of the face/non face recognition. Finally we emphasize the importance of the emotions as a mechanism to ensure the dynamical coupling between individuals allowing to learn more and more complex tasks.
---
paper_title: Visual transformations in gesture imitation: what you see is what you do

paper_content:

We propose an approach for a robot to imitate the gestures of a human demonstrator. Our framework consists solely of two components: a Sensory-Motor Map (SMM) and a View-Point Transformation (VPT). The SMM establishes an association between an arm image and the corresponding joint angles and it is learned by the system during a period of observation of its own gestures. The VPT is widely discussed in the psychology of visual perception and is used to transform the image of the demonstrator's arm to the so-called ego-centric image, as if the robot were observing its own arm. Different structures of the SMM and VPT are proposed in accordance with observations in human imitation. The whole system relies on monocular visual information and leads to a parsimonious architecture for learning by imitation. Real-time results are presented and discussed.
---
paper_title: Visual learning by imitation with motor representations

paper_content:

We propose a general architecture for action (mimicking) and program (gesture) level visual imitation. Action-level imitation involves two modules. The viewpoint transformation (VPT) performs a "rotation" to align the demonstrator's body to that of the learner. The visuo-motor map (VMM) maps this visual information to motor data. For program-level (gesture) imitation, there is an additional module that allows the system to recognize and generate its own interpretation of observed gestures to produce similar gestures/goals at a later stage. Besides the holistic approach to the problem, our approach differs from traditional work in i) the use of motor information for gesture recognition; ii) usage of context (e.g., object affordances) to focus the attention of the recognition system and reduce ambiguities, and iii) use iconic image representations for the hand, as opposed to fitting kinematic models to the video sequence. This approach is motivated by the finding of visuomotor neurons in the F5 area of the macaque brain that suggest that gesture recognition/imitation is performed in motor terms (mirror) and rely on the use of object affordances (canonical) to handle ambiguous actions. Our results show that this approach can outperform more conventional (e.g., pure visual) methods.
---
paper_title: Learning of Gestures by Imitation in a Humanoid Robot

paper_content:

In this chapter, we explore the issue of encoding, recognizing, generalizing and reproducing arbitrary gestures. We address one major and generic issue, namely how to discover the essence of a gesture, i.e. how to find a representation of the data that encapsulates only the key aspects of the gesture, and discards the intrinsic variability across people motions. The model is tested and validated in a humanoid robot, using kinematics data of human motion. In order for the robot to learn new skills by imitation, it must be endowed with the ability to generalize over multiple demonstrations. To achieve this, the robot must encode multivariate time-dependent data in an efficient way. Principal Component Analysis and Hidden Markov Models are used to reduce the dimensionality of the dataset and to extract the primitives of the motion. The model takes inspiration in a recent trend of research that aims at defining a formal mathematical framework for imitation learning. In particular, it stresses the fact that the observed elements of a demonstration, and the organization of these elements should be stochastically described to have a robust robotic application. It bears similarities with theoretical models of animal imitation, and offers at the same time a probabilistic description of the data, more suitable for a real-world application.
---
paper_title: Imitation without intentionality. Using string parsing to copy the organization of behaviour

paper_content:

A theory of imitation is proposed, string parsing, which separates the copying of behavioural organization by observation from an understanding of the cause of its effectiveness. In string parsing, recurring patterns in the visible stream of behaviour are detected and used to build a statistical sketch of the underlying hierarchical structure. This statistical sketch may in turn aid the subsequent comprehension of cause and effect. Three cases of social learning of relatively complex skills are examined, as potential cases of imitation by string parsing. Understanding the basic requirements for successful string parsing helps to resolve the conflict between mainly negative reports of imitation in experiments and more positive evidence from natural conditions. Since string parsing does not depend on comprehension of the intentions of other agents or the everyday physics of objects, separate tests of these abilities are needed even in animals shown to learn by imitation.
---
paper_title: Multiple paired forward and inverse models for motor control

paper_content:

Abstract Humans demonstrate a remarkable ability to generate accurate and appropriate motor behavior under many different and often uncertain environmental conditions. In this paper, we propose a modular approach to such motor learning and control. We review the behavioral evidence and benefits of modularity, and propose a new architecture based on multiple pairs of inverse (controller) and forward (predictor) models. Within each pair, the inverse and forward models are tightly coupled both during their acquisition, through motor learning, and use, during which the forward models determine the contribution of each inverse model's output to the final motor command. This architecture can simultaneously learn the multiple inverse models necessary for control as well as how to select the inverse models appropriate for a given environment. Finally, we describe specific predictions of the model, which can be tested experimentally.
---
paper_title: A unifying computational framework for motor control and social interaction

paper_content:

Recent empirical studies have implicated the use of the motor system during action observation, imitation and social interaction. In this paper, we explore the computational parallels between the processes that occur in motor control and in action observation, imitation, social interaction and theory of mind. In particular, we examine the extent to which motor commands acting on the body can be equated with communicative signals acting on other people and suggest that computational solutions for motor control may have been extended to the domain of social interaction.
---
paper_title: Primitive-Based Movement Classification for Humanoid Imitation

paper_content:

Motor control is a complex problem and imitation is a powerful mechanism for acquiring new motor skills. In this paper, we describe perceptuo-motor primitives, a biologically-inspired notion for a basis set of perceptual and motor routines. Primitives serve as a vocabulary for classifying and imitating observed human movements, and are derived from the imitator’s motor repertoire. We describe a model of imitation based on such primitives and demonstrate the feasibility of the model in a constrained implementation. We present approximate motion reconstruction generated from visually captured data of typically imitated tasks taken from aerobics, dancing, and athletics.
---
paper_title: Hidden Markov Model Approach to Skill Learning and Its Application to Telerobotics

paper_content:

In this paper, we discuss the problem of how human skill can be represented as a parametric model using a hidden Markov model (HMM), and how an HMM-based skill model can be used to learn human skill. HMM is feasible to characterize a doubly stochastic process--measurable action and immeasurable mental states--that is involved in the skill learning. We formulated the learning problem as a multidimensional HMM and developed a testbed for a variety of skill learning applications. Based on "the most likely performance" criterion, the best action sequence can be selected from all previously measured action data by modeling the skill as an HMM. The proposed method has been implemented in the teleoperation control of space station robot system, and some important implementation issues have been discussed. The method allows a robot to learn human skill in certain tasks and to improve motion performance.
---
paper_title: Skill acquisition from human demonstration using a hidden Markov model

paper_content:

A new approach to skill acquisition in assembly is proposed. An assembly skill is represented by a hybrid dynamic system where a discrete event controller models the skill at the task level. The output of the discrete event controller provides the reference commands for the underlying robot controller. This structure is naturally encoded by a hidden Markov model (HMM). The HMM parameters are obtained by training on sensory data from human demonstrations of the skill. Currently, assembly tasks have to be performed by human operators or by robots using expensive fixtures. Our approach transfers the assembly skill from an expert human operator to the robot, thus making it possible for a robot to perform assembly tasks without the use of expensive fixtures.
---
paper_title: Perception-Action Coupling via Imitation and Attention

paper_content:

When we interact with objects, we give them meaning, i.e. we show what they are potentially useful for. We believe that physical entities are anchored to perceptual representations, and through them to the actions that they ‘afford’. This paper brings an imitation mechanism and an attention system together computationally, with the aim of having a system that is capable of creating and maintaining these anchors. The integrated system is implemented on two different platforms: a simulated humanoid robot learning from another how to drink a glass of beer, and a simulated mobile robot learning from another how to follow walls.
---
paper_title: Hierarchical attentive multiple models for execution and recognition of actions

paper_content:

According to the motor theories of perception, the motor systems of an observer are actively involved in the perception of actions when these are performed by a demonstrator. In this paper we review our computational architecture, HAMMER (Hierarchical Attentive Multiple Models for Execution and Recognition), where the motor control systems of a robot are organised in a hierarchical, distributed manner, and can be used in the dual role of (a) competitively selecting and executing an action, and (b) perceiving it when performed by a demonstrator. We subsequently demonstrate that such an arrangement can provide a principled method for the top-down control of attention during action perception, resulting in significant performance gains. We assess these performance gains under a variety of resource allocation strategies. c 2006 Elsevier B.V. All rights reserved.
---
paper_title: Goal-directed imitation for robots: a bio-inspired approach to action understanding and skill learning

paper_content:

In this paper we present a robot control architecture for learning by imitation which takes inspiration from recent discoveries in action observation/execution experiments with humans and other primates. The architecture implements two basic processing principles: (1) imitation is primarily directed toward reproducing the outcome of an observed action sequence rather than reproducing the exact action means, and (2) the required capacity to understand the motor intention of another agent is based on motor simulation. The control architecture is validated in a robot system imitating in a goal-directed manner a grasping and placing sequence displayed by a human model. During imitation, skill transfer occurs by learning and representing appropriate goal-directed sequences of motor primitives. The robustness of the goal-directed organization of the controller is tested in the presence of incomplete visual information and changes in environmental constraints.
---
paper_title: Visual learning by imitation with motor representations

paper_content:

We propose a general architecture for action (mimicking) and program (gesture) level visual imitation. Action-level imitation involves two modules. The viewpoint transformation (VPT) performs a "rotation" to align the demonstrator's body to that of the learner. The visuo-motor map (VMM) maps this visual information to motor data. For program-level (gesture) imitation, there is an additional module that allows the system to recognize and generate its own interpretation of observed gestures to produce similar gestures/goals at a later stage. Besides the holistic approach to the problem, our approach differs from traditional work in i) the use of motor information for gesture recognition; ii) usage of context (e.g., object affordances) to focus the attention of the recognition system and reduce ambiguities, and iii) use iconic image representations for the hand, as opposed to fitting kinematic models to the video sequence. This approach is motivated by the finding of visuomotor neurons in the F5 area of the macaque brain that suggest that gesture recognition/imitation is performed in motor terms (mirror) and rely on the use of object affordances (canonical) to handle ambiguous actions. Our results show that this approach can outperform more conventional (e.g., pure visual) methods.
---
paper_title: Discovering optimal imitation strategies

paper_content:

This paper develops a general policy for learning the relevant features of an imitation task. We restrict our study to imitation of manipulative tasks or gestures. The imitation process is modeled as a hierarchical optimization system, which minimizes the discrepancy between two multi-dimensional datasets. To classify across manipulation strategies, we apply a probabilistic analysis to data in Cartesian and joint spaces. We determine a general metric that optimizes the policy of task reproduction, following strategy determination. The model successfully discovers strategies in six different imitative tasks and controls task reproduction by a full body humanoid robot. © 2004 Elsevier B.V. All rights reserved.
---
paper_title: Robotic imitation from human motion capture using gaussian processes

paper_content:

Programming by demonstration, also called "imitation learning," offers the possibility of flexible, easily modifiable robotic systems. Full-fledged robotic imitation learning comprises many difficult subtasks. However, we argue that, at its core, imitation learning reduces to a regression problem. We propose a two-step framework in which an imitating agent first performs a regression from a high-dimensional observation space to a low-dimensional latent variable space. In the second step, the agent performs a regression from the latent variable space to a high-dimensional space representing degrees of freedom of its motor system. We demonstrate the validity of the approach by learning to map motion capture data from human actors to a humanoid robot. We also contrast use of several low-dimensional latent variable spaces, each covering a subset of agents' degrees of freedom, with use of a single, higher-dimensional latent variable space. Our findings suggest that compositing several regression models together yields qualitatively better imitation results than using a single, more complex regression model
---
paper_title: Natural Actor-Critic

paper_content:

In this paper, we suggest a novel reinforcement learning architecture, the Natural Actor-Critic. The actor updates are achieved using stochastic policy gradients employing Amari's natural gradient approach, while the critic obtains both the natural policy gradient and additional parameters of a value function simultaneously by linear regression. We show that actor improvements with natural policy gradients are particularly appealing as these are independent of coordinate frame of the chosen policy representation, and can be estimated more efficiently than regular policy gradients. The critic makes use of a special basis function parameterization motivated by the policy-gradient compatible function approximation. We show that several well-known reinforcement learning methods such as the original Actor-Critic and Bradtke's Linear Quadratic Q-Learning are in fact Natural Actor-Critic algorithms. Empirical evaluations illustrate the effectiveness of our techniques in comparison to previous methods, and also demonstrate their applicability for learning control on an anthropomorphic robot arm.
---
paper_title: Interactive Policy Learning through Confidence-Based Autonomy

paper_content:

We present Confidence-Based Autonomy (CBA), an interactive algorithm for policy learning from demonstration. The CBA algorithm consists of two components which take advantage of the complimentary abilities of humans and computer agents. The first component, Confident Execution, enables the agent to identify states in which demonstration is required, to request a demonstration from the human teacher and to learn a policy based on the acquired data. The algorithm selects demonstrations based on a measure of action selection confidence, and our results show that using Confident Execution the agent requires fewer demonstrations to learn the policy than when demonstrations are selected by a human teacher. The second algorithmic component, Corrective Demonstration, enables the teacher to correct any mistakes made by the agent through additional demonstrations in order to improve the policy and future task performance. CBA and its individual components are compared and evaluated in a complex simulated driving domain. The complete CBA algorithm results in the best overall learning performance, successfully reproducing the behavior of the teacher while balancing the tradeoff between number of demonstrations and number of incorrect actions during learning.
---
paper_title: Learning movement primitives

paper_content:

This paper discusses a comprehensive framework for modular motor control based on a recently developed theory of dynamic movement primitives (DMP). DMPs are a formulation of movement primitives with autonomous nonlinear differential equations, whose time evolution creates smooth kinematic control policies. Model-based control theory is used to convert the outputs of these policies into motor commands. By means of coupling terms, on-line modifications can be incorporated into the time evolution of the differential equations, thus providing a rather flexible and reactive framework for motor planning and execution. The linear parameterization of DMPs lends itself naturally to supervised learning from demonstration. Moreover, the temporal, scale, and translation invariance of the differential equations with respect to these parameters provides a useful means for movement recognition. A novel reinforcement learning technique based on natural stochastic policy gradients allows a general approach of improving DMPs by trial and error learning with respect to almost arbitrary optimization criteria. We demonstrate the different ingredients of the DMP approach in various examples, involving skill learning from demonstration on the humanoid robot DB, and learning biped walking from demonstration in simulation, including self-improvement of the movement patterns towards energy efficiency through resonance tuning.
---
paper_title: Learning nonparametric models for probabilistic imitation

paper_content:

Learning by imitation represents an important mechanism for rapid acquisition of new behaviors in humans and robots. A critical requirement for learning by imitation is the ability to handle uncertainty arising from the observation process as well as the imitator's own dynamics and interactions with the environment. In this paper, we present a new probabilistic method for inferring imitative actions that takes into account both the observations of the teacher as well as the imitator's dynamics. Our key contribution is a nonparametric learning method which generalizes to systems with very different dynamics. Rather than relying on a known forward model of the dynamics, our approach learns a nonparametric forward model via exploration. Leveraging advances in approximate inference in graphical models, we show how the learned forward model can be directly used to plan an imitating sequence. We provide experimental results for two systems: a biomechanical model of the human arm and a 25-degrees-of-freedom humanoid robot. We demonstrate that the proposed method can be used to learn appropriate motor inputs to the model arm which imitates the desired movements. A second set of results demonstrates dynamically stable full-body imitation of a human teacher by the humanoid robot.
---
paper_title: Affordance, proper function, and the physical basis of perceived heaviness

paper_content:

Abstract The physical basis of perceived heaviness requires consideration of the haptic perceptual system's role in controlling actions (the system's proper function) and the relation of an object's inertial properties to properties of the human movement system (the object's affordance). We show that the mass of a wielded object and particular scalar variables calculated from the object's inertia tensor combine linearly in determining perceived heaviness. The tensor-derived scalars reflect the symmetry and volume of the corresponding inertia ellipsoid. These measures bear directly on the object's wieldability, that is, on the patterning and level of muscular forces required to move the object in a controlled fashion.
---
paper_title: A relational representation for procedural task knowledge

paper_content:

This paper proposes a methodology for learning joint probability estimates regarding the effect of sensorimotor features on the predicated quality of desired behavior. These relationships can then be used to choose actions that will most likely produce success. relational dependency networks are used to learn statistical models of procedural task knowledge. An example task expert for picking up objects is learned through actual experience with a humanoid robot. We believe that this approach is widely applicable and has great potential to allow a robot to autonomously determine which features in the world are salient and should be used to recommend policy for action.
---
paper_title: Changing affordances in stair climbing: the perception of maximum climbability in young and older adults

paper_content:

This experiment extended Warren's leg-length model by investigating the relevance of leg strength and joint flexibility on perceptual judgments of climbability. From a set of 8 stairs (riser heights: 38-91 cm), 24 older and 24 young adults were asked to identify the highest stair they could climb without using their hands or knees. Subjects then attempted to climb the selected stair. Tall and short young observers perceived similar action boundaries despite leg-length differences. Tall and short older adults had divergent action boundaries when a single-scale leg-length model was applied. A regression model that used flexibility and leg-strength measurements provided a better fit of the older adult data, indicating that models applying functional (kinematic and kinetic) criteria might be useful in describing lawful relationships between organisms and the environment. Language: en
---
paper_title: Towards a Real-Time Bayesian Imitation System for a Humanoid Robot

paper_content:

Imitation learning, or programming by demonstration (PbD), holds the promise of allowing robots to acquire skills from humans with domain-specific knowledge, who nonetheless are inexperienced at programming robots. We have prototyped a real-time, closed-loop system for teaching a humanoid robot to interact with objects in its environment. The system uses nonparametric Bayesian inference to determine an optimal action given a configuration of objects in the world and a desired future configuration. We describe our prototype implementation, show imitation of simple motor acts on a humanoid robot, and discuss extensions to the system
---
paper_title: Action recognition in the premotor cortex.

paper_content:

We recorded electrical activity from 532 neurons in the rostral part of inferior area 6 (area F5) of two macaque monkeys. Previous data had shown that neurons of this area discharge during goal-directed hand and mouth movements. We describe here the properties of a newly discovered set of F5 neurons (‘mirror neurons’, n = 92) all of which became active both when the monkey performed a given action and when it observed a similar action performed by the experimenter. Mirror neurons, in order to be visually triggered, required an interaction between the agent of the action and the object of it. The sight of the agent alone or of the object alone (three-dimensional objects, food) were ineffective. Hand and the mouth were by far the most effective agents. The actions most represented among those activating mirror neurons were grasping, manipulating and placing. In most mirror neurons (92%) there was a clear relation between the visual action they responded to and the motor response they coded. In ∼30% of mirror neurons the congruence was very strict and the effective observed and executed actions corresponded both in terms of general action (e.g. grasping) and in terms of the way in which that action was executed (e.g. precision grip). We conclude by proposing that mirror neurons form a system for matching observation and execution of motor actions. We discuss the possible role of this system in action recognition and, given the proposed homology between F5 and human Brocca’s region, we posit that a matching system, similar to that of mirror neurons exists in humans and could be involved in recognition of actions as well as phonetic gestures.
---
paper_title: To Afford or Not to Afford: A New Formalization of Affordances Toward Affordance-Based Robot Control

paper_content:

The concept of affordances was introduced by J. J. Gibson to eXplain how inherent "values" and "meanings" of things in the environment can be directly perceived and how this information can be linked to the action possibilities offered to the organism by the environment. Although introduced in psychology, the concept influenced studies in other fields ranging from human—computer interaction to autonomous robotics. In this article, we first introduce the concept of affordances as conceived by J. J. Gibson and review the use of the term in different fields, with particular emphasis on its use in autonomous robotics. Then, we summarize four of the major formalization proposals for the affordance term. We point out that there are three, not one, perspectives from which to view affordances and that much of the confusion regarding discussions on the concept has arisen from this. We propose a new formalism for affordances and discuss its implications for autonomous robot control. We report preliminary results obtained with robots and link them with these implications.
---
paper_title: Learning grasp strategies with partial shape information

paper_content:

We consider the problem of grasping novel objects in cluttered environments. If a full 3-d model of the scene were available, one could use the model to estimate the stability and robustness of different grasps (formalized as form/force-closure, etc); in practice, however, a robot facing a novel object will usually be able to perceive only the front (visible) faces of the object. In this paper, we propose an approach to grasping that estimates the stability of different grasps, given only noisy estimates of the shape of visible portions of an object, such as that obtained from a depth sensor. By combining this with a kinematic description of a robot arm and hand, our algorithm is able to compute a specific positioning of the robot's fingers so as to grasp an object. ::: ::: We test our algorithm on two robots (with very different arms/manipulators, including one with a multifingered hand). We report results on the task of grasping objects of significantly different shapes and appearances than ones in the training set, both in highly cluttered and in uncluttered environments. We also apply our algorithm to the problem of unloading items from a dishwasher.
---
paper_title: Learning grasping affordances from local visual descriptors

paper_content:

In this paper we study the learning of affordances through self-experimentation. We study the learning of local visual descriptors that anticipate the success of a given action executed upon an object. Consider, for instance, the case of grasping. Although graspable is a property of the whole object, the grasp action will only succeed if applied in the right part of the object. We propose an algorithm to learn local visual descriptors of good grasping points based on a set of trials performed by the robot. The method estimates the probability of a successful action (grasp) based on simple local features. Experimental results on a humanoid robot illustrate how our method is able to learn descriptors of good grasping points and to generalize to novel objects based on prior experience.
---
paper_title: Robotic Grasping of Novel Objects using Vision

paper_content:

We consider the problem of grasping novel objects, specifically objects that are being seen for the first time through vision. Grasping a previously unknown object, one for which a 3-d model is not available, is a challenging problem. Furthermore, even if given a model, one still has to decide where to grasp the object. We present a learning algorithm that neither requires nor tries to build a 3-d model of the object. Given two (or more) images of an object, our algorithm attempts to identify a few points in each image corresponding to good locations at which to grasp the object. This sparse set of points is then triangulated to obtain a 3-d location at which to attempt a grasp. This is in contrast to standard dense stereo, which tries to triangulate every single point in an image (and often fails to return a good 3-d model). Our algorithm for identifying grasp locations from an image is trained by means of supervised learning, using synthetic images for the training set. We demonstrate this approach on two robotic manipulation platforms. Our algorithm successfully grasps a wide variety of objects, such as plates, tape rolls, jugs, cellphones, keys, screwdrivers, staplers, a thick coil of wire, a strangely shaped power horn and others, none of which were seen in the training set. We also apply our method to the task of unloading items from dishwashers.
---
paper_title: Learning object-specific grasp affordance densities

paper_content:

This paper addresses the issue of learning and representing object grasp affordances, i.e. object-gripper relative configurations that lead to successful grasps. The purpose of grasp affordances is to organize and store the whole knowledge that an agent has about the grasping of an object, in order to facilitate reasoning on grasping solutions and their achievability. The affordance representation consists in a continuous probability density function defined on the 6D gripper pose space - 3D position and orientation -, within an object-relative reference frame. Grasp affordances are initially learned from various sources, e.g. from imitation or from visual cues, leading to grasp hypothesis densities. Grasp densities are attached to a learned 3D visual object model, and pose estimation of the visual model allows a robotic agent to execute samples from a grasp hypothesis density under various object poses. Grasp outcomes are used to learn grasp empirical densities, i.e. grasps that have been confirmed through experience. We show the result of learning grasp hypothesis densities from both imitation and visual cues, and present grasp empirical densities learned from physical experience by a robot.
---
paper_title: Learning about objects through action - initial steps towards artificial cognition

paper_content:

Within the field of Neuro Robotics we are driven primarily by the desire to understand how humans and animals live and grow and solve every day's problems. To this aim we adopted a "learn by doing" approach by building artificial systems, e.g. robots that not only look like human beings but also represent a model of some brain process. They should, ideally, behave and interact like human beings (being situated). The main emphasis in robotics has been on systems that act as a reaction to an external stimulus (e.g. tracking, reaching), rather than as a result of an internal drive to explore or "understand" the environment. We think it is now appropriate to try to move from acting, in the sense explained above, to "understanding". As a starting point we addressed the problem of learning about the effects and consequences of self-generated actions. How does the robot learn how to pull an object toward itself or to push it away? How does the robot learn that spherical objects roll while a cube only slides if pushed? Interacting with objects is important because it implicitly explores object representation, event understanding, and can provide definition of object-hood that could not be grasped with a mere passive observation of the world. Further, learning to understand what one's own body can do is an essential step toward learning by imitation. In this view two actions are similar not only if their kinematics and dynamics are similar but rather if the effects on the external world are the same. Along this line of research we discuss some recent experiments performed at the AI-Lab at MIT and at the LIRA-Lab at the University of Genova on COG and Babybot respectively. We show how the humanoid robots can learn how to poke and prod objects to obtain a consistently repeatable effect (e.g. sliding in a given direction), to help visual segmentation, and to interpret a poking action performed by a human manipulator.
---
paper_title: Learning and performing place-based mobile manipulation

paper_content:

What it means for an object to be ‘within reach’ depends very much on the morphology and skills of a robot. In this paper, we enable a mobile manipulation robot to learn a concept of PLACE from which successful manipulation is possible through trial-and-error interaction with the environment. Due to this developmental approach, PLACE is very much grounded in observed experience, and takes the hardware and skills of the robot into account. During task-execution, this model is used to determine optimal grasp places in a least-commitment approach. This PLACE takes into account uncertainties in both robot and target object positions, and leads to more robust behavior.
---
paper_title: Learning Predictive Features in Affordance based Robotic Perception Systems

paper_content:

This work is about the relevance of Gibson's concept of affordances [1] for visual perception in interactive and autonomous robotic systems. In extension to existing functional views on visual feature representations, we identify the importance of learning in perceptual cueing for the anticipation of opportunities for interaction of robotic agents. We investigate how the originally defined representational concept for the perception of affordances - in terms of using either optical flow or heuristically determined 3D features of perceptual entities - should be generalized to using arbitrary visual feature representations. In this context we demonstrate the learning of causal relationships between visual cues and predictable interactions, and emphasize on a novel framework for cueing and hypothesis verification of affordances that could play an important role in future robot control architectures. We argue that affordance based perception should enable systems to react to environment stimuli both more efficient and autonomous, and provide a potential to plan on the basis of responses to more complex perceptual configurations. We verify the concept with a concrete implementation applying state-of-the-art visual descriptors and regions of interest within a simulated robot scenario and prove that these features were successfully selected for predicting opportunities of robot interaction.
---
paper_title: Self-valuing learning and generalization with application in visually guided grasping of complex objects

paper_content:

Abstract For programming by demonstration and for reinforcement learning, the learned skills need appropriate representations for generalization and possibilities for further improvements by the robot itself. We present a self-valuing learning technique which is capable of learning how to grasp unfamiliar objects and generalize the learned abilities. The learning system consists of two components which distinguish between local and global quality criteria for grasp points. The local criteria are not object-specific while the global criteria cover physical properties of each object. In this case we present a generalization method of the learning parameters based on a tree distance model for the medial axis transformations. The system is self-valuing, i.e. it rates its actions by evaluating sensory information and the usage of image processing techniques. This learning system has been implemented in a real robot assembly system equipped with hand-cameras and force/torque sensors. Both the theory and the experiments have shown it ability to grasp a wide range of objects and to apply pre-learned knowledge to new objects.
---
paper_title: Behavior-Grounded Representation of Tool Affordances

paper_content:

This paper introduces a novel approach to representing and learning tool affordances by a robot. The tool representation described here uses a behavior-based approach to ground the tool affordances in the behavioral repertoire of the robot. The representation is learned during a behavioral babbling stage in which the robot randomly chooses different exploratory behaviors, applies them to the tool, and observes their effects on environmental objects. The paper shows how the autonomously learned affordance representation can be used to solve tool-using tasks by dynamically sequencing the exploratory behaviors based on their expected outcomes. The quality of the learned representation was tested on extension-of-reach tool-using tasks.
---
paper_title: Towards a Real-Time Bayesian Imitation System for a Humanoid Robot

paper_content:

Imitation learning, or programming by demonstration (PbD), holds the promise of allowing robots to acquire skills from humans with domain-specific knowledge, who nonetheless are inexperienced at programming robots. We have prototyped a real-time, closed-loop system for teaching a humanoid robot to interact with objects in its environment. The system uses nonparametric Bayesian inference to determine an optimal action given a configuration of objects in the world and a desired future configuration. We describe our prototype implementation, show imitation of simple motor acts on a humanoid robot, and discuss extensions to the system
---
paper_title: Goal-directed imitation for robots: a bio-inspired approach to action understanding and skill learning

paper_content:

In this paper we present a robot control architecture for learning by imitation which takes inspiration from recent discoveries in action observation/execution experiments with humans and other primates. The architecture implements two basic processing principles: (1) imitation is primarily directed toward reproducing the outcome of an observed action sequence rather than reproducing the exact action means, and (2) the required capacity to understand the motor intention of another agent is based on motor simulation. The control architecture is validated in a robot system imitating in a goal-directed manner a grasping and placing sequence displayed by a human model. During imitation, skill transfer occurs by learning and representing appropriate goal-directed sequences of motor primitives. The robustness of the goal-directed organization of the controller is tested in the presence of incomplete visual information and changes in environmental constraints.
---
paper_title: Learning Object Affordances: From Sensory--Motor Coordination to Imitation

paper_content:

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.
---
paper_title: Learning Bayesian networks: the combination of knowledge and statistical data

paper_content:

We describe scoring metrics for learning Bayesian networks from a combination of user knowledge and statistical data. We identify two important properties of metrics, which we call event equivalence and parameter modularity. These properties have been mostly ignored, but when combined, greatly simplify the encoding of a user's prior knowledge. In particular, a user can express his knowledge--for the most part--as a single prior Bayesian network for the domain.
---
paper_title: An anthropomorphic robot torso for imitation: design and experiments

paper_content:

We describe the design of an anthropomorphic robot, combining a binocular head, an arm and a hand, for research in visuomotor coordination and learning by imitation. Our goal was to produce a system resembling the human arm-hand kinematics as closely as possible, while keeping it simple and relatively low-cost. We present mechanical details, kinematics and sensors together with a discussion of the main design options. We present results with human-arm coordination, as well as imitation of a human demonstrator, in real time.
---
paper_title: Learning Object Affordances: From Sensory--Motor Coordination to Imitation

paper_content:

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.
---
paper_title: A Developmental Roadmap for Learning by Imitation in Robots

paper_content:

In this paper, we present a strategy whereby a robot acquires the capability to learn by imitation following a developmental pathway consisting on three levels: 1) sensory-motor coordination; 2) world interaction; and 3) imitation. With these stages, the system is able to learn tasks by imitating human demonstrators. We describe results of the different developmental stages, involving perceptual and motor skills, implemented in our humanoid robot, Baltazar. At each stage, the system's attention is drawn toward different entities: its own body and, later on, objects and people. Our main contributions are the general architecture and the implementation of all the necessary modules until imitation capabilities are eventually acquired by the robot. Also, several other contributions are made at each level: learning of sensory-motor maps for redundant robots, a novel method for learning how to grasp objects, and a framework for learning task description from observation for program-level imitation. Finally, vision is used extensively as the sole sensing modality (sometimes in a simplified setting) avoiding the need for special data-acquisition hardware
---
paper_title: A model for inferring the intention in imitation tasks

paper_content:

Robot imitation comprises a number of hard problems, one of the most difficult problems is the extraction of intention from a demonstration. A demonstration can almost always be interpreted in different ways, making it impossible for the imitating robot to find out what exactly it is that was demonstrated. We first attempt to set out the problem of intention reading. Next, we offer a computational model which implements a solution to intention reading. Our model needs repeated interactions between the demonstrator and the imitator. Through keeping a score about which interactions where successful, the imitating robot gradually builds a model which ?understands? what the intent is of the demonstrator.
---
paper_title: Discovering optimal imitation strategies

paper_content:

This paper develops a general policy for learning the relevant features of an imitation task. We restrict our study to imitation of manipulative tasks or gestures. The imitation process is modeled as a hierarchical optimization system, which minimizes the discrepancy between two multi-dimensional datasets. To classify across manipulation strategies, we apply a probabilistic analysis to data in Cartesian and joint spaces. We determine a general metric that optimizes the policy of task reproduction, following strategy determination. The model successfully discovers strategies in six different imitative tasks and controls task reproduction by a full body humanoid robot. © 2004 Elsevier B.V. All rights reserved.
---
paper_title: Inferring the goals of a nonhuman agent

paper_content:

Johnson, Slaughter, and Carey [Dev. Sci. 1 (1998) 233.] used infants' ability to follow the ‘gaze’ of novel objects to claim that infants' recognition of mentalistic agents is not isomorphic with person recognition but rather based on a set of nonarbitrary object recognition cues including the presence of a face and the ability to interact contingently with other agents. The current studies extend these findings with data based on infant imitation and the production of communicative gestures. The first study replicated Meltzoff's [Dev. Psychol. 31 (1995) 838.] reenactment of goals paradigm with a novel, nonhuman agent. Fifteen-month-olds were found to reenact both the completed and uncompleted/unseen goals of a novel object that had a face and interacted contingently with the experimenter and infant. Concurrently, infants directed many communicative gestures at the object. A second study excluded the possibility that the communicative gestures apparently directed at the object in Study 1 were in fact imitations of the experimenter's own behavior. The suggestion that novel, nonhuman objects are capable of eliciting such divergent behaviors as gaze-following, goal reenactment, and communicative gestures from infants, supports the claim that all of these behaviors are mediated by a central conceptual notion of mentalistic being, at least by the ages studied, and that that concept is not isomorphic with the concept person.
---
paper_title: Maximum entropy inverse reinforcement learning

paper_content:

Recent research has shown the benefit of framing problems of imitation learning as solutions to Markov Decision Problems. This approach reduces learning to the problem of recovering a utility function that makes the behavior induced by a near-optimal policy closely mimic demonstrated behavior. In this work, we develop a probabilistic approach based on the principle of maximum entropy. Our approach provides a well-defined, globally normalized distribution over decision sequences, while providing the same performance guarantees as existing methods. ::: ::: We develop our technique in the context of modeling real-world navigation and driving behaviors where collected data is inherently noisy and imperfect. Our probabilistic approach enables modeling of route preferences as well as a powerful new approach to inferring destinations and routes based on partial trajectories.
---
paper_title: A unified framework for imitation-like behaviors

paper_content:

In this paper, we combine the formal methods from re- inforcement learning with the paradigm of imitation learning. The extension of the reinforcement learning framework to integrate the information provided by an expert (demonstrator) has the important advantage of allowing a clear decrease of the time necessary to learn certain robotic tasks. Hence, learning by imitation can be interpreted as a mechanism for fast skill transfer. Another contribution of this paper consists in showing that our formalism is able to model dif- ferent types of imitation-learning that are described in the biological literature. It thus unifies in the same abstract model what used to be addressed as separate behavioral patterns. We illustrate the applica- tion of these methods in simulation and with a real robot.
---
paper_title: Affordance-based imitation learning in robots

paper_content:

In this paper we build an imitation learning algorithm for a humanoid robot on top of a general world model provided by learned object affordances. We consider that the robot has previously learned a task independent affordance-based model of its interaction with the world. This model is used to recognize the demonstration by another agent (a human) and infer the task to be learned. We discuss several important problems that arise in this combined framework, such as the influence of an inaccurate model in the recognition of the demonstration. We illustrate the ideas in the paper with some experimental results obtained with a real robot.
---
paper_title: Hierarchical attentive multiple models for execution and recognition of actions

paper_content:

According to the motor theories of perception, the motor systems of an observer are actively involved in the perception of actions when these are performed by a demonstrator. In this paper we review our computational architecture, HAMMER (Hierarchical Attentive Multiple Models for Execution and Recognition), where the motor control systems of a robot are organised in a hierarchical, distributed manner, and can be used in the dual role of (a) competitively selecting and executing an action, and (b) perceiving it when performed by a demonstrator. We subsequently demonstrate that such an arrangement can provide a principled method for the top-down control of attention during action perception, resulting in significant performance gains. We assess these performance gains under a variety of resource allocation strategies. c 2006 Elsevier B.V. All rights reserved.
---
paper_title: Doing the Right Thing: Infants' Selection of Actions to Imitate From Observed Event Sequences

paper_content:

Two studies were conducted to investigate how 14- to 16-month-old infants select actions to imitate from the stream of events. In each study, an experimenter demonstrated two actions leading to an interesting effect. Aspects of the first action were manipulated and whether infants performed this action when given the objects was observed. In both studies, infants were more likely to imitate the first action when it was physically necessary to generate the effect, and in Study 2 they were also more likely to imitate the action when it was socially cued. It seems that infants' own knowledge of space and causality as well as their sensitivity to others' social signals both contribute to their tendency to imitate actions.
---
paper_title: Goal-Based Imitation as Probabilistic Inference over Graphical Models

paper_content:

Humans are extremely adept at learning new skills by imitating the actions of others. A progression of imitative abilities has been observed in children, ranging from imitation of simple body movements to goal-based imitation based on inferring intent. In this paper, we show that the problem of goal-based imitation can be formulated as one of inferring goals and selecting actions using a learned probabilistic graphical model of the environment. We first describe algorithms for planning actions to achieve a goal state using probabilistic inference. We then describe how planning can be used to bootstrap the learning of goal-dependent policies by utilizing feedback from the environment. The resulting graphical model is then shown to be powerful enough to allow goal-based imitation. Using a simple maze navigation task, we illustrate how an agent can infer the goals of an observed teacher and imitate the teacher even when the goals are uncertain and the demonstration is incomplete.
---
paper_title: Bayesian models of human action understanding

paper_content:

We present a Bayesian framework for explaining how people reason about and predict the actions of an intentional agent, based on observing its behavior. Action-understanding is cast as a problem of inverting a probabilistic generative model, which assumes that agents tend to act rationally in order to achieve their goals given the constraints of their environment. Working in a simple sprite-world domain, we show how this model can be used to infer the goal of an agent and predict how the agent will act in novel situations or when environmental constraints change. The model provides a qualitative account of several kinds of inferences that preverbal infants have been shown to perform, and also fits quantitative predictions that adult observers make in a new experiment.
---
paper_title: Goal-Based Imitation as Probabilistic Inference over Graphical Models

paper_content:

Humans are extremely adept at learning new skills by imitating the actions of others. A progression of imitative abilities has been observed in children, ranging from imitation of simple body movements to goal-based imitation based on inferring intent. In this paper, we show that the problem of goal-based imitation can be formulated as one of inferring goals and selecting actions using a learned probabilistic graphical model of the environment. We first describe algorithms for planning actions to achieve a goal state using probabilistic inference. We then describe how planning can be used to bootstrap the learning of goal-dependent policies by utilizing feedback from the environment. The resulting graphical model is then shown to be powerful enough to allow goal-based imitation. Using a simple maze navigation task, we illustrate how an agent can infer the goals of an observed teacher and imitate the teacher even when the goals are uncertain and the demonstration is incomplete.
---
paper_title: Bayesian models of human action understanding

paper_content:

We present a Bayesian framework for explaining how people reason about and predict the actions of an intentional agent, based on observing its behavior. Action-understanding is cast as a problem of inverting a probabilistic generative model, which assumes that agents tend to act rationally in order to achieve their goals given the constraints of their environment. Working in a simple sprite-world domain, we show how this model can be used to infer the goal of an agent and predict how the agent will act in novel situations or when environmental constraints change. The model provides a qualitative account of several kinds of inferences that preverbal infants have been shown to perform, and also fits quantitative predictions that adult observers make in a new experiment.
---
paper_title: Imitation with ALICE: Learning to Imitate Corresponding Actions across Dissimilar Embodiments

paper_content:

Imitation is a powerful mechanism whereby knowledge may be transferred between agents (both biological and artificial). Key problems on the topic of imitation have emerged in various areas close to artificial intelligence, including the cognitive and social sciences, animal behavior, robotics, human-computer interaction, embodied intelligence, software engineering, programming by example and machine learning. Artificial systems used to study imitation can both test models of imitation derived from observational or neurobiological data on imitation in animals and then apply them to different kinds of nonbiological systems ranging from robots to software agents. A crucial problem in imitation is the correspondence problem, mapping action sequences of the demonstrator and the imitator agent. This problem becomes particularly obvious when the two agents do not share the same embodiment and affordances. This paper describes a new general imitation mechanism called ALICE (action learning for imitation via correspondence between embodiments) that specifically addresses the correspondence problem. The mechanism is implemented and its efficacy illustrated on the "chessworld" testbed that was created to study imitation from an agent-based perspective, i.e., by a particular agent in a particular environment.
---
paper_title: Affordance-based imitation learning in robots

paper_content:

In this paper we build an imitation learning algorithm for a humanoid robot on top of a general world model provided by learned object affordances. We consider that the robot has previously learned a task independent affordance-based model of its interaction with the world. This model is used to recognize the demonstration by another agent (a human) and infer the task to be learned. We discuss several important problems that arise in this combined framework, such as the influence of an inaccurate model in the recognition of the demonstration. We illustrate the ideas in the paper with some experimental results obtained with a real robot.
---
paper_title: A unified framework for imitation-like behaviors

paper_content:

In this paper, we combine the formal methods from re- inforcement learning with the paradigm of imitation learning. The extension of the reinforcement learning framework to integrate the information provided by an expert (demonstrator) has the important advantage of allowing a clear decrease of the time necessary to learn certain robotic tasks. Hence, learning by imitation can be interpreted as a mechanism for fast skill transfer. Another contribution of this paper consists in showing that our formalism is able to model dif- ferent types of imitation-learning that are described in the biological literature. It thus unifies in the same abstract model what used to be addressed as separate behavioral patterns. We illustrate the applica- tion of these methods in simulation and with a real robot.
---
paper_title: Apprenticeship Learning using Inverse Reinforcement Learning and Gradient Methods

paper_content:

In this paper we propose a novel gradient algorithm to learn a policy from an expert's observed behavior assuming that the expert behaves optimally with respect to some unknown reward function of a Markovian Decision Problem. The algorithm's aim is to find a reward function such that the resulting optimal policy matches well the expert's observed behavior. The main difficulty is that the mapping from the parameters to policies is both nonsmooth and highly redundant. Resorting to sub differentials solves the first difficulty, while the second one is overcome by computing natural gradients. We tested the proposed method in two artificial domains and found it to be more reliable and efficient than some previous methods.
---
paper_title: Apprenticeship learning using linear programming

paper_content:

In apprenticeship learning, the goal is to learn a policy in a Markov decision process that is at least as good as a policy demonstrated by an expert. The difficulty arises in that the MDP's true reward function is assumed to be unknown. We show how to frame apprenticeship learning as a linear programming problem, and show that using an off-the-shelf LP solver to solve this problem results in a substantial improvement in running time over existing methods---up to two orders of magnitude faster in our experiments. Additionally, our approach produces stationary policies, while all existing methods for apprenticeship learning output policies that are "mixed", i.e. randomized combinations of stationary policies. The technique used is general enough to convert any mixed policy to a stationary policy.
---
paper_title: Bayesian Inverse Reinforcement Learning

paper_content:

Inverse Reinforcement Learning (IRL) is the problem of learning the reward function underlying a Markov Decision Process given the dynamics of the system and the behaviour of an expert. IRL is motivated by situations where knowledge of the rewards is a goal by itself (as in preference elicitation) and by the task of apprenticeship learning (learning policies from an expert). In this paper we show how to combine prior knowledge and evidence from the expert's actions to derive a probability distribution over the space of reward functions. We present efficient algorithms that find solutions for the reward learning and apprenticeship learning tasks that generalize well over these distributions. Experimental results show strong improvement for our methods over previous heuristic-based approaches.
---
paper_title: 'obsessed with goals' : Functions and mechanisms of teleological interpretation of actions in humans

paper_content:

Humans show a strong and early inclination to interpret observed behaviours of others as goal-directed actions. We identify two main epistemic functions that this ‘teleological obsession’ serves: on-line prediction and social learning. We show how teleological action interpretations can serve these functions by drawing on two kinds of inference (‘action-to-goal’ or ‘goal-to-action’), and argue that both types of teleological inference constitute inverse problems that can only be solved by further assumptions. We pinpoint the assumptions that the three currently proposed mechanisms of goal attribution (action-effect associations, simulation procedures, and teleological reasoning) imply, and contrast them with the functions they are supposed to fulfil. We argue that while action-effect associations and simulation procedures are generally well suited to serve on-line action monitoring and prediction, social learning of new means actions and artefact functions requires the inferential productivity of teleological reasoning.
---
paper_title: Action, State and Effect Metrics for Robot Imitation

paper_content:

This paper addresses the problem of body mapping in robotic imitation where the demonstrator and imitator may not share the same embodiment (degrees of freedom (DOFs), body morphology, constraints, affordances and so on). Body mappings are formalized using a unified (linear) approach via correspondence matrices, which allow one to capture partial, mirror symmetric, one-to-one, one-to-many, many-to-one and many-to-many associations between various DOFs across dissimilar embodiments. We show how metrics for matching state and action aspects of behaviour can be mathematically determined by such correspondence mappings, which may serve to guide a robotic imitator. The approach is illustrated in a number of examples, using agents described by simple kinematic models and different types of correspondence mappings. Also, focusing on aspects of displacement and orientation of manipulated objects, a selection of metrics are presented, towards a characterization of the space of effect metrics.
---
paper_title: Infant imitation after a 1-week delay: Long-term memory for novel acts and multiple stimuli

paper_content:

Deferred imitation after a l-week delay was examined in 14-month-old infants. Six actions, each using a different object, were demonstrated to each infant. One of the six actions was a novel behavior that had a zero probability of occurrence in spontaneous play. In the imitation condition, infants observed the demonstration but were not allowed to touch the objects, thus preventing any immediate imitation. After the 1-week delay, infants returned to the laboratory and their imitation of the adult's previous actions was scored. Infants in the imitation condition produced significantly more of the target actions than infants in control groups who were not exposed to the modeling; there was also strong evidence for the imitation of the novel act. From a cognitive perspective deferred imitation provides a means of assessing recall memory and representation in children. From a socialdevelopmental viewpoint the findings illustrate that the behavioral repertoire of infants and their knowledge about objects can expand as a result of seeing the actions of others. Developmental theories differ in the extent to which imitation is viewed as making a contribution to the growth of a child's behavioral repertoire. Piaget (1952, 1954) tended to emphasize individual discovery and to minimize social imitation as a mechanism for acquiring new behaviors. Conversely, Bandura (1969, 1986) highlighted imitation as a channel for learning in school-age children, and Meltzoff (1985, 1988a, 1988b, in press) has emphasized the powerful influence of imitation in infancy. Regardless of the differing perspectives on the function of imitation, there is broad agreement that infants' ability to perform imitation after a delay, and especially deferred imitation of novel behaviors, is a milestone achievement both from a cognitive and a social perspective (Bandura, 1986; Flavell, 1985; Meltzoff, 1988b; Piaget, 1962). The acquisition of new behaviors through observation, particularly when they can delay production until significantly later, potentially provides both an avenue for social learning and a nonverbal measure of memory and representation. Although imitation after a delay has been actively investigated in school-age children (Bandura, Grusec, & Menlove, 1966; Bandura & Waiters, 1963), there have been only a few such experimental investigations in infants. McCall, Parke, and Kavanaugh (1977) reported imitation after a 1-day delay in 26month-old subjects. Meltzoff (1985) found deferred imitation of a simple action after a 1-day delay in 14-month-olds. Recently, it has been reported that infants as young as 9 months
---
paper_title: An anthropomorphic robot torso for imitation: design and experiments

paper_content:

We describe the design of an anthropomorphic robot, combining a binocular head, an arm and a hand, for research in visuomotor coordination and learning by imitation. Our goal was to produce a system resembling the human arm-hand kinematics as closely as possible, while keeping it simple and relatively low-cost. We present mechanical details, kinematics and sensors together with a discussion of the main design options. We present results with human-arm coordination, as well as imitation of a human demonstrator, in real time.
---
paper_title: Teaching collaborative multi-robot tasks through demonstration

paper_content:

Humanoid robots working alongside humans in everyday environments is a long standing goal of the robotics community. To achieve this goal, methods for developing new robot behaviors that are intuitive and accessible to non-programmers are required. In this paper, we present a demonstration-based method for teaching distributed autonomous robots to coordinate their actions and perform collaborative multi-robot tasks. Within the presented framework, each robot learns an individual policy from teacher demonstrations using a confidence-based algorithm. Based on this learning approach, we contribute three techniques for teaching multi-robot coordination using different information sharing strategies. We evaluate and compare these approaches by teaching two Sony QRIO humanoid robots to perform three collaborative ball sorting tasks.
---
paper_title: Parallel and distributed neural models of the ideomotor principle: An investigation of imitative cortical pathways

paper_content:

Humans' capacity to imitate has been extensively investigated through a wide-range of behavioral and developmental studies. Yet, despite the huge amount of phenomenological evidence gathered, we are still unable to relate this behavioral data to any specific neural substrate. In this paper, we investigate how principles from psychology can be the result of neural computations and therefore attempt to bridge the gap between monkey neurophysiology and human behavioral data, and hence between these two complementary disciplines. Specifically, we address the principle of ideomotor compatibility, by which 'observing the movements of others influences the quality of one's own performance' and develop two neural models which account for a set of related behavioral studies [Brass, M., Bekkering, H., Wohlschlager, A., & Prinz, W. (2000). Compatibility between observed and executed finger movements: comparing symbolic, spatial and imitative cues. Brain and Cognition 44, 124-143]. We show that the ideomotor effect could be the result of two distinct cognitive pathways, which can be modeled by means of biologically plausible neural architectures. Furthermore, we propose a novel behavioral experiment to confirm or refute either of the two model pathways.
---
paper_title: Imitation: A Means to Enhance Learning of a Synthetic Proto-Language in an Autonomous Robot.

paper_content:

This chapter contains sections titled: Introduction, Robotic Models of Imitation, The Role of Imitation in Language Learning, Robotic Experiments, Experiments, Discussion, Conclusion, Acknowledgments, Notes, References
---
paper_title: Active learning for reward estimation in inverse reinforcement learning

paper_content:

Inverse reinforcement learning addresses the general problem of recovering a reward function from samples of a policy provided by an expert/demonstrator. In this paper, we introduce active learning for inverse reinforcement learning. We propose an algorithm that allows the agent to query the demonstrator for samples at specific states, instead of relying only on samples provided at "arbitrary" states. The purpose of our algorithm is to estimate the reward function with similar accuracy as other methods from the literature while reducing the amount of policy samples required from the expert. We also discuss the use of our algorithm in higher dimensional problems, using both Monte Carlo and gradient methods. We present illustrative results of our algorithm in several simulated examples of different complexities.
---
paper_title: The progress drive hypothesis: an interpretation of early imitation

paper_content:

Children seem to have a natural tendency to imitate and their interest for particular kinds of imitative behaviour varies greatly with the infant’s age. We argue that different forms of children’s early imitation may be the result of an intrinsic motivation system driving the infant into situations of maximal learning progress. We present a computational model showing how an agent can learn to focus on “progress niches”, situations neither completely predictable nor too difficult to predict given its anticipation capabilities. The existence of such a drive could explain why certain types of imitative behaviour are produced by children at a certain age, and how discriminations between self, others and objects gradually appear.
---
paper_title: A unified framework for imitation-like behaviors

paper_content:

In this paper, we combine the formal methods from re- inforcement learning with the paradigm of imitation learning. The extension of the reinforcement learning framework to integrate the information provided by an expert (demonstrator) has the important advantage of allowing a clear decrease of the time necessary to learn certain robotic tasks. Hence, learning by imitation can be interpreted as a mechanism for fast skill transfer. Another contribution of this paper consists in showing that our formalism is able to model dif- ferent types of imitation-learning that are described in the biological literature. It thus unifies in the same abstract model what used to be addressed as separate behavioral patterns. We illustrate the applica- tion of these methods in simulation and with a real robot.
---
paper_title: Social Learning Mechanisms Compared in a Simple Environment

paper_content:

Social learning can be adaptive, but little is known about the underlying mechanisms. Many researchers have focused on imitation but this may have led to simpler mechanisms being underestimated. We demonstrate in simulation that imitative learning is not always the best strategy for a group-living animal, and that the effectiveness of any such strategy will depend on details of the environment and the animal's lifestyle. We show that observations of behavioural convergence or "traditions" might suggest effective social learning, but are meaningless considered alone.
---
paper_title: A Neurocomputational Model of an Imitation Deficit Following Brain Lesion

paper_content:

This paper investigates the neural mechanisms of visuo-motor imitation in humans through convergent evidence from neuroscience. In particular, we consider a deficit in imitation following callosal brain lesion, based on the rational that looking at how imitation is impaired can unveil its underlying neural principles. We ground the functional architecture and information flow of our model in brain imaging studies and use findings from monkey brain neurophysiological studies to drive the choice of implementation of our processing modules. Our neural model of visuo-motor imitation is based on self-organizing maps with associated activities. Patterns of impairment of the model, realized by adding uncertainty in the transfer of information between the networks, account for the scores found in a clinical examination of imitation [1]. The model also allows several interesting predictions.
---
paper_title: Active Imitation Learning

paper_content:

Imitation learning, also called learning by watching or programming by demonstration, has emerged as a means of accelerating many reinforcement learning tasks. Previous work has shown the value of imitation in domains where a single mentor demonstrates execution of a known optimal policy for the benefit of a learning agent. We consider the more general scenario of learning from mentors who are themselves agents seeking to maximize their own rewards. We propose a new algorithm based on the concept of transferable utility for ensuring that an observer agent can learn efficiently in the context of a selfish, not necessarily helpful, mentor. We also address the questions of when an imitative agent should request help from a mentor, and when the mentor can be expected to acknowledge a request for help. In analogy with other types of active learning, we call the proposed approach active imitation learning.
---
paper_title: Attention coupling as a prerequisite for social interaction

paper_content:

This paper proposes, "attention coupling", that is spatio-temporal coordination of each other's attention, as a prerequisite for human-robot social interaction, where the human interactant attributes mental states to the robot, and possibly vice versa. As a realization of attention coupling we implemented on our robots the capability of eye-contact (mutually looking into each other's eyes) and joint attention (looking at a shared target together). Observation of the interaction with human babies/children showed that the robots with the attention coupling capability facilitated in the babies/children social behavior, including showing, giving, and verbal interactions like asking questions.
---
paper_title: Robots as Isolators or Mediators for Children with Autism A Cautionary Tale

paper_content:

The discussion presented in this paper is part of our investigation in the Aurora project into the potential use of robots as therapeutic or educational ‘toys’ specifically for use by children with autism. The paper raises some cautions concerning social isolation and stereotypical behaviour frequently exhibited in children with autism. We present some examples taken from trials with the robots where the children exhibit such behaviour, and discuss possible ways of ensuring not to reinforce stereotypical behaviour and a tendency to social isolation in the children. Especially, we point out an avenue of robots becoming social mediators (mediating contact between children and other children or adults). The paper exemplifies interaction where social behaviour was directed at the robot which raises awareness of the goal of the research, namely to help the children to increase their social interaction skills with other people and not simply create relationships with a ‘social’ robot which would isolate the children from other humans even further.
---
paper_title: Maximum margin planning

paper_content:

Imitation learning of sequential, goal-directed behavior by standard supervised techniques is often difficult. We frame learning such behaviors as a maximum margin structured prediction problem over a space of policies. In this approach, we learn mappings from features to cost so an optimal policy in an MDP with these cost mimics the expert's behavior. Further, we demonstrate a simple, provably efficient approach to structured maximum margin learning, based on the subgradient method, that leverages existing fast algorithms for inference. Although the technique is general, it is particularly relevant in problems where A* and dynamic programming approaches make learning policies tractable in problems beyond the limitations of a QP formulation. We demonstrate our approach applied to route planning for outdoor mobile robots, where the behavior a designer wishes a planner to execute is often clear, while specifying cost functions that engender this behavior is a much more difficult task.
---
paper_title: Reinforcement learning with limited reinforcement: using Bayes risk for active learning in POMDPs

paper_content:

Partially Observable Markov Decision Processes (POMDPs) have succeeded in planning domains that require balancing actions that increase an agent's knowledge and actions that increase an agent's reward. Unfortunately, most POMDPs are defined with a large number of parameters which are difficult to specify only from domain knowledge. In this paper, we present an approximation approach that allows us to treat the POMDP model parameters as additional hidden state in a "model-uncertainty" POMDP. Coupled with model-directed queries, our planner actively learns good policies. We demonstrate our approach on several POMDP problems.
---
paper_title: On the development of intention understanding for joint action tasks

paper_content:

Our everyday, common sense ability to discern the intentions of others' from their motions is fundamental for a successful cooperation in joint action tasks. In this paper we address in a modeling study the question of how the ability to understand complex goal-directed action sequences may develop during learning and practice. The model architecture reflects recent neurophysiological findings that suggest the existence of chains of mirror neurons associated with specific goals. These chains may be activated by external events to simulate the consequences of observed actions. Using the mathematical framework of dynamical neural fields to model the dynamics of different neural populations representing goals, action means and contextual cues, we show that such chains may develop based on a local, Hebbian learning rule. We validate the functionality of the learned model in a joint action task in which an observer robot infers the intention of a partner to chose a complementary action sequence.
---
paper_title: A developmental approach accelerates learning of joint attention

paper_content:

This paper argues how robot learning can be accelerated by a developmental approach that changes the capability of the learning robot (robot development) and/or the complexity of the environment (environmental development). "Robot development" means that the perceptual, cognitive, and behavioral capabilities of the learning robot change from immature to mature states in accordance with learning process. On the other hand, "environmental development" means that the complexity of the environmental structure changes from simple to complicated ones in accordance with learning process. Then, the developmental approaches can be categorized according to the relationships between what does develop and what does make the development. A case study of developmental robot learning for joint attention in which a caregiver can be regarded as a part of the environment is given to show how it can accelerate the learning. Finally, future work and discussion are given.
---
paper_title: A unifying computational framework for motor control and social interaction

paper_content:

Recent empirical studies have implicated the use of the motor system during action observation, imitation and social interaction. In this paper, we explore the computational parallels between the processes that occur in motor control and in action observation, imitation, social interaction and theory of mind. In particular, we examine the extent to which motor commands acting on the body can be equated with communicative signals acting on other people and suggest that computational solutions for motor control may have been extended to the domain of social interaction.
---

Title: Abstraction Levels for Robotic Imitation: Overview and Computational Approaches

Section 1: Introduction
Description 1: This section introduces the topic of robotic imitation, the various cognitive capabilities required, and the survey's focus on self-modeling in cognitive processes.

Section 2: Why Learn by Imitation?
Description 2: This section discusses the importance and advantages of learning by imitation for robots, paralleling the way human infants learn through imitation.

Section 3: What Is Imitation?
Description 3: This segment defines imitation in both biological and robotic contexts, exploring various terminologies and classifications within social learning mechanisms.

Section 4: Organization of the Chapter
Description 4: This section outlines the chapter’s structure, detailing the main perspectives and methods of imitation that will be discussed.

Section 5: Imitation in Natural Systems
Description 5: This section details insights from psychology and neurophysiology on imitation in biological systems, highlighting the complexity of imitation.

Section 6: Neurophysiology
Description 6: This part explores neurophysiological findings related to action understanding and their implications for robotic learning approaches.

Section 7: Motor Theories of Perception
Description 7: This section discusses theories suggesting that the motor system is involved in perception, including the existence of forward and backward models in the brain.

Section 8: Mirror and Canonical Neurons
Description 8: Focuses on the discovery and roles of mirror and canonical neurons in action recognition and imitation.

Section 9: Reasoning Processes
Description 9: This section examines the necessity of reasoning mechanisms for understanding complex and novel observed actions.

Section 10: Psychology
Description 10: Discusses behavioral psychology studies on imitative behaviors in children and chimpanzees, including task constraints and imperfect knowledge.

Section 11: Imitation in Artificial Systems
Description 11: Outlines the complexities and challenges of developing robotic systems capable of learning by imitation.

Section 12: Data Acquisition
Description 12: Discusses methods and tools for acquiring perceptual data necessary for robotic imitation.

Section 13: Mapping of Perceptual Variables and Body Correspondence
Description 13: Explores techniques for mapping external inputs to internal motor representations, important for perception-to-action mapping and body correspondence.

Section 14: Goal/Intention Inference
Description 14: This section delves into the reasoning processes needed for a robot to infer the intentions behind observed actions.

Section 15: The Role of Self-Observation
Description 15: Examines the importance of self-observation for robots to learn about their own body and the world dynamics.

Section 16: Imitating by Motor Resonance
Description 16: Presents methods that learn by demonstration through mapping and generalizing state-action trajectories.

Section 17: Visual Transformations
Description 17: Discusses the role of viewpoint transformations in mapping demonstrator actions to learner actions.

Section 18: Mimicking Behaviors and Automatic Imitation
Description 18: Explores approaches that use simple motor resonance to transfer observed behaviors to robots.

Section 19: Imitation through Motor Primitives
Description 19: Investigates methods that recognize and sequence observed motor primitives for more complex interactions.

Section 20: Learning of New Task Solutions
Description 20: Analyzes approaches where robots use imitation to learn new actions and improve their performance through reinforcement learning and other strategies.

Section 21: Object Mediated Imitation
Description 21: Discusses imitation from an object-centric perspective, focusing on the higher-level representations of actions and effects on objects.

Section 22: Affordances as Perception-Action Maps
Description 22: Examines approaches that learn mappings from object features to changes in these features to facilitate emulation.

Section 23: Affordances as Dynamical Models
Description 23: Explores modeling the dynamical system composed of agents and objects for emulation.

Section 24: Bayesian Networks as Models for Affordances
Description 24: Investigates using Bayesian networks to model statistical relations between actions, objects, and their effects for affordance learning.

Section 25: Experiments
Description 25: Provides experimental results demonstrating the application of the Bayesian network model for affordances in robotic platforms.

Section 26: Imitating by Inferring the Goal
Description 26: Discusses approaches that abstract learning by imitation to the task level, focusing on inferring the goals behind observed actions.

Section 27: Goal Inference from Demonstration
Description 27: Examines methods for inferring the task to be learned from observed demonstrations, using various computational approaches.

Section 28: Inverse Reinforcement Learning as Goal Inference
Description 28: Introduces modeling methods for task inference from demonstrations through inverse reinforcement learning.

Section 29: Environment Model
Description 29: Describes the formalism used to model interaction with the environment and the learning of new action combinations.

Section 30: Bayesian Recovery of the Task Description
Description 30: Details the Bayesian approach to estimating task descriptions from observed demonstrations.

Section 31: A Model for Social Learning Mechanisms
Description 31: Presents a comprehensive model for social learning mechanisms incorporating multiple sources of information.

Section 32: Modeling Imitation in Humans
Description 32: Applies the imitation learning model to human social learning experiments and assesses the learner's behavior change.

Section 33: Robot Learning by Imitation
Description 33: Demonstrates an application of the imitation learning model in a robotic platform performing a task.

Section 34: Other Imitation Settings
Description 34: Explores other settings and applications of imitation learning, including social interactions, multiagent scenarios, and biological insights.

Section 35: Discussion
Description 35: Summarizes the insights from the chapter, discussing the implications, open questions, and future research directions in imitation learning.