Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines
---
paper_title: ‘ ‘ Social ’ ’ robots are psychological agents for infants : A test of gaze following

paper_content:

Gaze following is a key component of human social cognition. Gaze following directs attention to areas of high information value and accelerates social, causal, and cultural learning. An issue for both robotic and infant learning is whose gaze to follow. The hypothesis tested in this study is that infants use information derived from an entity's interactions with other agents as evidence about whether that entity is a perceiver. A robot was programmed so that it could engage in communicative, imitative exchanges with an adult experimenter. Infants who saw the robot act in this social-communicative fashion were more likely to follow its line of regard than those without such experience. Infants use prior experience with the robot's interactions as evidence that the robot is a psychological agent that can see. Infants want to look at what the robot is seeing, and thus shift their visual attention to the external target.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: Expectancies for social contingency in 2-month-olds.

paper_content:

In a classic report, Murray and Trevarthen (Emotional regulation of interaction between two-month-olds and their mothers. In T. Field & N. Fox (Eds), Social perception in infants (pp. 101–125). Norwood, NJ: Ablex, 1985) suggested that 6- to 12-week-olds were sensitive to contingency in maternal stimulation because they became upset during the replay of their mothers’ behavior during an initial period of live interaction over closed circuit TV. However, a number of concerns were raised about the procedure and it was argued that infants may simply have become increasingly fussy over time. To clarify this very important issue, we adopted a three-condition design (live–replay–live) and modified Murray and Trevarthen’s procedure by inserting a seamless shift from the initially live interaction period to a replay period of good maternal behavior. This was followed by a seamless shift back to a subsequent live interaction. Two-month-old smiling and gaze decreased while frowning and mouth closures increased during the replay period relative to the initial live interaction period, replicating Murray and Trevarthen’s results. Fussiness over time did not account for these results because seven of the ten babies increased their positive behaviors and decreased negative behaviors during the second live interaction period, and there was a significant increase in their visual attention to their mothers, thus suggesting a renewed interest in the previously noncontingent partner.
---
paper_title: The development of dialogue co-ordination skills in schoolchildren

paper_content:

Abstract This paper explores certain ways in which dialogue co-ordination skills develop in schoolchildren between 7 and 12 years of age. The analysis is based on a large corpus (from 80 pairs of speakers) of task-oriented dialogues elicited using Garrod and Anderson's (1987) cooperative maze game procedure. As in previous work with adult conversants, our analysis concentrates on how the speakers establish local description “languages” or schemes to describe their locations on the maze. The results indicate that there is strong pressure for speakers of all ages to co-ordinate on common description schemes. However, more detailed analysis of the dialogues suggests that there are two underlying principles of co-ordination operating. While the youngest conversants (7 to 8-year-olds) show evidence of superficial co-ordination processes establishing a limited common dialogue lexicon and particular common description scheme only the older speakers show evidence of deep co-ordination processes aimed at directly ...
---
paper_title: Saying what you mean in dialogue: A study in conceptual and semantic co-ordination

paper_content:

Abstract This paper explores how conversants co-ordinate their use and interpretation of language in a restricted context. It revolves around the analysis of the spatial descriptions which emerge during the course of 56 dialogues, elicited in the laboratory using a specially designed computer maze game. Two types of analysis are reported. The first is a semantic analysis of the various types of description, which indicates how pairs of speakers develop different language schemes associated with different mental models of the maze configuration. The second analysis concerns how the communicants co-ordinate in developing their description schemes. The results from this study would suggest that language processing in dialogue may be governed by local principles of interaction which have received little attention in the psychological and linguistic literature to date.
---
paper_title: Premotor cortex and the recognition of motor actions

paper_content:

Abstract In area F5 of the monkey premotor cortex there are neurons that discharge both when the monkey performs an action and when he observes a similar action made by another monkey or by the experimenter. We report here some of the properties of these ‘mirror’ neurons and we propose that their activity ‘represents’ the observed action. We posit, then, that this motor representation is at the basis of the understanding of motor events. Finally, on the basis of some recent data showing that, in man, the observation of motor actions activate the posterior part of inferior frontal gyrus, we suggest that the development of the lateral verbal communication system in man derives from a more ancient communication system based on recognition of hand and face gestures.
---
paper_title: Facial EMG reactions to facial expressions: A case of facial emotional contagion?

paper_content:

The purpose of this study is to explore whether subjects exposed to stimuli of facial expressions respond with facial electromyographic (EMG) reactions consistent with the hypothesis that facial ex ...
---
paper_title: Toward a mechanistic psychology of dialogue

paper_content:

Traditional mechanistic accounts of language processing derive almost entirely from the study of monologue. Yet, the most natural and basic form of language use is dialogue. As a result, these accounts may only offer limited theories of the mechanisms that un- derlie language processing in general. We propose a mechanistic account of dialogue, the interactive alignment account, and use it to de- rive a number of predictions about basic language processes. The account assumes that, in dialogue, the linguistic representations em- ployed by the interlocutors become aligned at many levels, as a result of a largely automatic process. This process greatly simplifies production and comprehension in dialogue. After considering the evidence for the interactive alignment model, we concentrate on three aspects of processing that follow from it. It makes use of a simple interactive inference mechanism, enables the development of local di- alogue routines that greatly simplify language processing, and explains the origins of self-monitoring in production. We consider the need for a grammatical framework that is designed to deal with language in dialogue rather than monologue, and discuss a range of implica- tions of the account.
---
paper_title: Why is conversation so easy?

paper_content:

Traditional accounts of language processing suggest that monologue – presenting and listening to speeches – should be more straightforward than dialogue – holding a conversation. This is clearly not the case. We argue that conversation is easy because of an interactive processing mechanism that leads to the alignment of linguistic representations between partners. Interactive alignment occurs via automatic alignment channels that are functionally similar to the automatic links between perception and behaviour (the so-called perception–behaviour expressway) proposed in recent accounts of social interaction. We conclude that humans are ‘designed’ for dialogue rather than monologue. Whereas many people find it difficult to present a speech or even listen to one, we are all very good at talking to each other. This might seem a rather obvious and banal observation, but from a cognitive point of view the apparent ease of conversation is paradoxical. The range and complexity of the information that is required in monologue (preparing and listening to speeches) is much less than is required in dialogue (holding a conversation). In this article we suggest that dialogue processing is easy because it takes advantage of a processing mechanism that we call ‘interactive alignment’. We argue that interactive alignment is automatic and reflects the fact that humans are designed for dialogue rather than monologue. We show how research in social cognition points to other similar automatic alignment mechanisms. Problems posed by dialogue
---
paper_title: TALK AND SILENCE SEQUENCES IN INFORMAL CONVERSATIONS III: INTERSPEAKER INFLUENCE

paper_content:

Literature on the structure of two-person conversations has consistently found that partners become more similar in mean duration of pauses and switching pauses over the course of interaction. Evidence on influence in vocalization duration is primarily negative. No direct evidence of interspeaker influence on a moment-to-moment basis is available. In this study 12 dyadic conversations are analyzed for interspeaker influence, with the use of time series regression procedures. Two versions of the data are presented: a probability summary including the probability of breaking mutual silences, continuing simultaneous talk, and continuing talking alone and a turn summary including vocalization, pause, and switching pause duration. Results show that (1) moment-to-moment influences are present in both versions of the data, (2) these influences differ from dyad to dyad, (3) the influences are both positive (matching) and negative (compensating), and (4) the magnitude of interspeaker influence on a temporal basis is small but detectable. In addition, there are overall tendencies to match in switching pause, probability of continued simultaneous speech, and probability of breaking mutual silences. Dyads show both compensation and matching on vocalization duration and pause-related variables. Implications of these data for past and future explanation of social interaction are explored.
---
paper_title: Accommodation theory: Communication, context, and consequence.

paper_content:

Introduction When academic theorizing addresses everyday communication phenomena, there are losses as well as gains. Research may, selectively or otherwise, partially represent the full subtlety of contextualized interaction. Methodological constraints may impose their own selectivity, so that we tend to access the accessible and learn what is most readily learnable. The real-time nature of programmatic research will reflect epistemological shifts and disciplinary development. It is altogether likely that academic and lay versions of the phenomena themselves and their boundaries will not perfectly mirror each other at any one point. On the other hand, research can discover regularities within communicative interchanges and identify, and perhaps even predict, contextual configurations that relate systematically to them. If it is amenable to methodological triangulation upon data and research questions, and if it incorporates within its own activities a mechanism for building cumulatively on empirical insights, communication research can begin to impose order on the uncertainty that interaction presents to us. More particularly, research that addresses the contexts as much as the behaviors of talk can tease out the ordering – motivational, strategic, behavioral, attributional, and evaluative – that interactants themselves impose upon their own communication experiences, and the ways in which the social practices of talk both are constrained by and themselves constrain goals, identities, and social structures. In the case of “accommodation theory,” the focus of the present collection, we have a research program that has developed over more than a dozen years, undergoing many extensions and elaborations, as an account of contextual processes impinging on sociolinguistic code, style, and strategy selections.
---
paper_title: Dominance and accommodation in the conversational behaviours of same- and mixed-gender dyads.

paper_content:

Among the first writers to posit a relationship between the language women use and their social condition was the linguist Robin Lakoff (1975; see also Key, 1973, who defined a ‘female register’ that serves as a marker for women’s subordinate social status and reinforces the attributions of dependency, incompetence and timidity that are part of the female stereotype. From this perspective, differences in male and female speech are a reflection of the general dominance of males in society, a view that has been termed the ma/e dominance ~y~~~~~~~~ (Thorne and Henley, 1975). Although Lakoff’s observations were not based on systematic examination of actual speech corpora, there is evidence to indicate that some of the characteristics of the hypothesized female register are differentials distributed in male and female speech, at least in some situations (Crosby and Nyquist, 1977; Smith, 1985; Ruke-Dravina, 1952; but see also Brouwer et at., 1979).
---
paper_title: Using Nonconscious Behavioral Mimicry to Create Affiliation and Rapport

paper_content:

Nonconscious behavioral mimicry occurs when a person unwittingly imitates the behaviors of another person. This mimicry has been attributed to a direct link between perceiving a behavior and performing that same behavior. The current experiments explored whether having a goal to affiliate augments the tendency to mimic the behaviors of interaction partners. Experiment 1 demonstrated that having an affiliation goal increases nonconscious mimicry, and Experiment 2 further supported this proposition by demonstrating that people who have unsuccessfully attempted to affiliate in an interaction subsequently exhibit more mimicry than those who have not experienced such a failure. Results suggest that behavioral mimicry may be part of a person's repertoire of behaviors, used nonconsciously, when there is a desire to create rapport.
---
paper_title: Conceptual pacts and lexical choice in conversation

paper_content:

When people in conversation refer repeatedly to the same object, they come to use the same terms. This phenomenon, called lexical entrainment, has several possible explanations. Ahistorical accounts appeal only to the informativeness and availability of terms and to the current salience of the object's features. Historical accounts appeal in addition to the recency and frequency of past references and to partner-specific conceptualizations of the object that people achieve interactively. Evidence from 3 experiments favors a historical account and suggests that when speakers refer to an object, they are proposing a conceptualization of it, a proposal their addressees may or may not agree to. Once they do establish a shared conceptualization, a conceptual pact, they appeal to it in later references even when they could use simpler references. Over time, speakers simplify conceptual pacts and, when necessary, abandon them for new conceptualizations.
---
paper_title: The chameleon effect : The perception-behavior link and social interaction

paper_content:

The chameleon effect refers to nonconscious mimicry of the postures, mannerisms, facial expressions, and other behaviors of one's interaction partners, such that one's behavior passively and unintentionally changes to match that of others in one's current social environment. The authors suggest that the mechanism involved is Has. perception-behavior link, the recently documented finding (e.g., J. A. Bargh, M. Chen, & L. Burrows, 1996) that the mere perception of another's behavior automatically increases the likelihood of engaging in that behavior oneself. Experiment 1 showed that the motor behavior of participants unintentionally matched that of strangers with whom they worked on a task. Experiment 2 had confederates mimic the posture and movements of participants and showed that mimicry facilitates the smoothness of interactions and increases liking between interaction partners. Experiment 3 showed that dispositionally empathic individuals exhibit the chameleon effect to a greater extent than do other people.
---
paper_title: ACCENT MOBILITY: A MODEL AND SOME DATA

paper_content:

0. 0. Labov (1966) recorded formal interviews of about o,,:. and a h,.od lf1 hour' s duration in the homes of 81 informants from the Lowe r E ast Side of New York. The interviews were conducted to deterrnine whetllher ch..nges ij c:li.te.4t would affect pronunciation patterns of the interviewees. He ypo .h!lesi zed that. formal contexts of social interaction would require a greater reli ance on prestigious variants of pronunciati on by the speal ke r, than 1e: ss for n Ia.l .0. ntexts of the discourse. A substantial portion of the intervi ews was c.orns.ide redt as-; (4lictiting
---
paper_title: Multimodal embodied mimicry in interaction

paper_content:

Nonverbal behavior plays an important role in human-human interaction. One particular kind of nonverbal behavior is mimicry. Behavioral mimicry supports harmonious relationships in social interaction through creating affiliation, rapport, and liking between partners. Affective computing that employs mimicry knowledge and that is able to predict how mimicry affects social situations and relations can find immediate application in human-computer interaction to improve interaction. In this short paper we survey and discuss mimicry issues that are important from that point of view: application in human-computer interaction. We designed experiments to collect mimicry data. Some preliminary analysis of the data is presented.
---
paper_title: Dyadic synchrony: Its structure and function in children’s development ☆

paper_content:

In this review we examine empirical and theoretical work in three eras—infancy, toddlerhood, and early childhood—and for each era describe the structure of dyadic synchrony in interactions involving children and their caregivers, as well as offer speculation about its developmental function for the child. We review divergent literatures dealing with synchrony-related constructs which, together, suggest that although the structure and function of synchrony change throughout the course of early development, the ability to achieve synchrony may represent a crucial developmental achievement for significant dyadic relationships, one that facilitates social, emotional, and cognitive growth for the child.
---
paper_title: Neonate Movement Is Synchronized with Adult Speech: Interactional Participation and Language Acquisition

paper_content:

As early as the first day of life, the human neonate moves in precise and sustained segments of movement that are synchronous with the articulated structure of adult speech. These observations suggest a view of development of the infant as a participant at the outset in multiple forms of interactional organization, rather than as an isolate.
---
paper_title: Development Through Life: A Psychosocial Approach

paper_content:

1. THE DEVELOPMENT THROUGH LIFE PERSPECTIVE. Assumptions of the Text. A Psychosocial Approach: The Interaction of the Biological, Psychological, and Societal Systems. The Scientific Process. 2. PSYCHOSOCIAL THEORY. What Is a Theory? The PSYCHOSOCIAL Perspective. Basic Concepts of Psychosocial Theory. Evaluation of Psychosocial Theory. The Life Span. Chapter Summary. References. Case Study: Rose. 3. MAJOR THEORIESFOR UNDERSTANDING CHANGE. The Theory of Evolution. Cognitive Development Theory. Theories of Learning. Cultural Theory. Social Role Theory. Systems Theory. Chapter Summary. References. 4. THE PERIOD OF PREGNANCY AND PRENATAL DEVELOPMENT. Genetics and Development. Genetic Sources of Individuality. Genetic Technology and Psychosocial Evolution. Evaluating the Impact of Heredity on Behavior. Normal Fetal Development. The Birth Process. The Mother, the Fetus, and the Psychosocial Environment. Applied topic: Abortion. Summary. References. 5. INFANCY (FIRST 24 MONTHS). Newborns. The Development of Sensory/Perceptual and Motor Functions. Attachment. Sensorimotor Intelligence and Early Causal Schemes. Understanding the Nature of Objects and Creating Categories. Emotional Development. The Psychosocial Crisis: Trust versus Mistrust. The Central Process for Resolving the Crisis: Mutuality with the Caregiver. The Prime Adaptive Ego Quality and the Core Pathology. Applied Topic: The Role of Parents. Chapter Summary. References. 6. TODDLERHOOD (AGES 2 AND 3). Developmental Tasks. The Psychosocial Crisis: Autonomy versus Shame and Doubt. The Central Process: Imitation. The Prime Adaptive Ego Quality and the Core Pathology. The Impact of Poverty on Psychosocial Development in Toddlerhood. Applied Topic: Day Care. Chapter Summary. References. 7. EARLY SCHOOL AGE (4 TO 6 YEARS). Developmental Tasks. The Psychosocial Crisis: Initiative Versus Guilt. The Central Process: Identification. The Prime Adaptive Ego Quality and the Core Pathology. Applied Topic: School Readiness. Chapter Summary. References. 8. MIDDLE CHILDHOOD (6-12 YEARS). Developmental Tasks. The Psychosocial Crisis: Industry versus Inferiority. The Central Process: Education. The Prime Adaptive Ego Quality and the Core Pathology. Applied Topic: Violence in the Lives of Children. Chapter Summary. References. 9. EARLY ADOLESCENCE (12-18 YEARS). Developmental Tasks. The Psychosocial Crisis: Group Identity versus Alienation. The Central Process: Peer Pressure. The Prime Adaptive Ego Quality and the Core Pathology. Applied Topic: Adolescent Alcohol and Drug Use. Chapter Summary.
---
paper_title: Synchrony - a core concept for a constructivist approach to psychotherapy

paper_content:

Synchrony is a pervasive concept relevant to diverse domains in physics, biology, and the social sciences. This article reviews some of the evidence both from natural and human systems. Our main focus is on the role of synchrony in psychotherapy research. Its association with empathy, rapport and the therapeutic relationship has been pointed out repeatedly, yet close evaluation of empirical studies suggests that the evidence remains inconclusive. Particularly in naturalistic studies, research employing synchrony as a specific measure is still lacking. We suggest a new approach to studying synchrony in psychotherapies under field conditions: Image-differencing is a video-based algorithm quantifying the amount of movement in pre-selected regions. Our methodology can be compared to the approach from the seminal work of Condon & Ogston (1966). The statistical analysis we employ detects synchrony on a global level, irrespective of the specific body parts moving. Synchrony thus defined is considered a general measure of coordination of interacting individuals. Exemplary data show the feasibility of this approach and its potential contribution to psychotherapy research.
---
paper_title: Are there biological programming effects for psychological development?: Findings from a study of Romanian adoptees

paper_content:

Associations between experiences and outcomes could be due to (a) continuation of adversity or (b) organismic changes, including experience-expectant and experience-adaptive developmental programming. The adoption into British families of children who had been reared in profoundly depriving institutions in Romania presented an opportunity to test mechanisms. Romanian children reared from infancy in very depriving institutions for periods up to 42 months were compared with 52 nondeprived UK-born children placed into adoptive families before the age of 6 months. The results at 6 years of age showed substantial normal cognitive and social functioning after the provision of family rearing but also major persistent deficits in a substantial minority. The pattern of findings suggests some form of early biological programming or neural damage stemming from institutional deprivation, but the heterogeneity in outcome indicates that the effects are not deterministic. During the 1950s to 1970s, strong claims were made regarding the supposed permanent effects of early adverse experiences and also about the importance of critical periods in development that required particular experiences to occur during a narrow time frame for normal development to proceed (Bowlby, 1951; Klaus & Kennell, 1976; Pilling & Kellmer-Pringle, 1978; World Health Organization Expert Committee on Mental Health, 1951). Both claims failed to be supported by empirical research findings, at least with respect to their strong form (Bateson, 1966; Bornstein, 1987; Rutter, 1981; Sluckin, 1973), and the concepts went out of fashion. It came to be accepted that experiences at all ages could be influential and that the extent to which the effects of adverse experiences did or did not persist depended in large part on whether the early disadvantage or deprivation was followed by later disadvantage or deprivation (Clarke & Clarke, 1976, 2000). During the 1990s, there was a reemergence of these earlier claims in a slightly different form, supposedly supported by neuroscience findings on early brain development and on the ways in which such development is sculpted and shaped by early experiences (see Bruer, 1999, for a critique). In this article, we seek first to clarify the relevant theoretical notions and then to use our data set on children reared in extremely depriving Romanian orphanages, who were subsequently adopted into United Kingdom (UK) families at ages ranging from infancy to 31U2 years and then followed up at the ages of 4 and 6 years (they have recently been reassessed at 11–12 years), in order to test alternative causative models. The sample provided a particularly striking example of a “natural experiment” (see Rutter, Pickles, Murray, & Eaves, 2001) in which there was a sharp discontinuity between early and later rearing environments and in which the change was extremely sudden (and thus easy to time exactly) and also involved an unusually radical shift from a profound and pervasive institutional deprivation to somewhat above-average rearing circumstances in a low-risk family setting. The key question is whether, given the high quality of the later environment, there were any persisting sequelae and, if there were, to what they might be due.
---
paper_title: Early language acquisition: cracking the speech code

paper_content:

Infants learn language with remarkable speed, but how they do it remains a mystery. New data show that infants use computational strategies to detect the statistical and prosodic patterns in language input, and that this leads to the discovery of phonemes and words. Social interaction with another human being affects speech learning in a way that resembles communicative learning in songbirds. The brain's commitment to the statistical and prosodic patterns that are experienced early in life might help to explain the long-standing puzzle of why infants are better language learners than adults. Successful learning by infants, as well as constraints on that learning, are changing theories of language acquisition.
---
paper_title: Movement coordination in social interaction: Some examples described

paper_content:

Abstract Sound films of social interaction are analysed and some detailed descriptions of the interrelations of the movements of speakers ans listeners are given. These descriptions provide further examples of ‘interactional synchrony’, first described by Condon and Ogston (1966). It is found that the flow of movement in the listener may be rhythmically coordinated with the speech and movements of the speaker. It is also shown how the way in which individuals may be in synchrony with one another can vary, and that these variations are related to their respective roles in the interaction. Some of the implications of the phenomenon of interactional synchrony are discussed.
---
paper_title: Dyadic synchrony: Its structure and function in children’s development ☆

paper_content:

In this review we examine empirical and theoretical work in three eras—infancy, toddlerhood, and early childhood—and for each era describe the structure of dyadic synchrony in interactions involving children and their caregivers, as well as offer speculation about its developmental function for the child. We review divergent literatures dealing with synchrony-related constructs which, together, suggest that although the structure and function of synchrony change throughout the course of early development, the ability to achieve synchrony may represent a crucial developmental achievement for significant dyadic relationships, one that facilitates social, emotional, and cognitive growth for the child.
---
paper_title: Disturbances of attachment and parental psychopathology in early childhood.

paper_content:

As the field of attachment has expanded over the past four decades, the perturbations in the relational context which give rise to disturbances of attachment are increasingly, though by no means conclusively, understood. In Part I, this article reviews the historical and current state of research regarding normative attachment classification, the diagnosis of Reactive Attachment Disorder, and the proposed categories of Secure Base Distortions and Disrupted Attachment Disorder. In Part II, the article explores the role of parental psychopathology and the manner in which disturbed caregiver self-regulation leads to disturbances in the mutual regulation between caregiver and infant. The question of the relationship between particular types of maternal pathology and particular forms of attachment disturbance is examined through recent research on the association between maternal PTSD, Atypical Maternal Behavior, and child scores on the Disturbances of Attachment Interview (DAI). The authors present original research findings to support that the presence and severity of maternal violence-related PTSD were significantly associated with secure base distortion in a community pediatrics sample of 76 mothers and preschool-age children. Clinical implications and recommendations for treatment of attachment disturbances conclude the article.
---
paper_title: Infant Intersubjectivity: Research, Theory, and Clinical Applications

paper_content:

We review research evidence on the emergence and development of active ‘‘self-and-other’’ awareness in infancy, and examine the importance of its motives and emotions to mental health practice with children. This relates to how communication begins and develops in infancy, how it influences the individual subject’s movement, perception, and learning, and how the infant’s biologically grounded self-regulation of internal state and self-conscious purposefulness is sustained through active engagement with sympathetic others. Mutual selfother-consciousness is found to play the lead role in developing a child’s cooperative intelligence for cultural learning and language. A variety of preconceptions have animated rival research traditions investigating infant communication and cognition. We distinguish the concept of ‘‘intersubjectivity’’, and outline the history of its use in developmental research. Thetransforming body and brain of a humanindividual grows inactive engagement with an environment of human factors—organic at first, then psychological or inter-mental. Adaptive, human-responsive processes are generated first by interneuronal activity within the developing brain as formation of the human embryo is regulated in a support-system of maternal tissues. Neural structures are further elaborated with the benefit of intra-uterine stimuli in the foetus, then supported in the rapidly growing forebrain and cerebellum of the young child by experience of the intuitive responses of parents and other human companions. We focus particularly on intrinsic patterns and processes in pre-natal and post-natal brain maturation that anticipate psychosocial support in infancy. The operation of an intrinsic motive formation (IMF) that developed in the core of the brain before birth is evident in the tightly integrated intermodal sensory-motor coordination of a newborn infant’s orienting to stimuli and preferential learning of human signals, by the temporal coherence and intrinsic rhythms of infant behaviour, especially in communication, and neonates’ extraordinary capacities for reactive and evocative imitation. The correct functioning of this integrated neural motivating system is found to be essential to the development of both the infant’s purposeful consciousness and his or her ability to cooperate with other persons’ actions and interests, and to learn from them. The relevance of infants’ inherent intersubjectivity to major child mental health issues is highlighted by examining selected areas of clinical concern. We review recent findings on postnatal depression, prematurity, autism, ADHD, specific language impairments, and central auditory processing deficits, and comment on the ecacy of interventions that aim to support intrinsic motives for intersubjective communication when these are not developing normally.
---
paper_title: Social interaction shapes babbling: Testing parallels between birdsong and speech

paper_content:

Birdsong is considered a model of human speech development at behavioral and neural levels. Few direct tests of the proposed analogs exist, however. Here we test a mechanism of phonological development in human infants that is based on social shaping, a selective learning process first documented in songbirds. By manipulating mothers' reactions to their 8-month-old infants' vocalizations, we demonstrate that phonological features of babbling are sensitive to nonimitative social stimulation. Contingent, but not noncontingent, maternal behavior facilitates more complex and mature vocal behavior. Changes in vocalizations persist after the manipulation. The data show that human infants use social feedback, facilitating immediate transitions in vocal behavior. Social interaction creates rapid shifts to developmentally more advanced sounds. These transitions mirror the normal development of speech, supporting the predictions of the avian social shaping model. These data provide strong support for a parallel in function between vocal precursors of songbirds and infants. Because imitation is usually considered the mechanism for vocal learning in both taxa, the findings introduce social shaping as a general process underlying the development of speech and song.
---
paper_title: Development Through Life: A Psychosocial Approach

paper_content:

1. THE DEVELOPMENT THROUGH LIFE PERSPECTIVE. Assumptions of the Text. A Psychosocial Approach: The Interaction of the Biological, Psychological, and Societal Systems. The Scientific Process. 2. PSYCHOSOCIAL THEORY. What Is a Theory? The PSYCHOSOCIAL Perspective. Basic Concepts of Psychosocial Theory. Evaluation of Psychosocial Theory. The Life Span. Chapter Summary. References. Case Study: Rose. 3. MAJOR THEORIESFOR UNDERSTANDING CHANGE. The Theory of Evolution. Cognitive Development Theory. Theories of Learning. Cultural Theory. Social Role Theory. Systems Theory. Chapter Summary. References. 4. THE PERIOD OF PREGNANCY AND PRENATAL DEVELOPMENT. Genetics and Development. Genetic Sources of Individuality. Genetic Technology and Psychosocial Evolution. Evaluating the Impact of Heredity on Behavior. Normal Fetal Development. The Birth Process. The Mother, the Fetus, and the Psychosocial Environment. Applied topic: Abortion. Summary. References. 5. INFANCY (FIRST 24 MONTHS). Newborns. The Development of Sensory/Perceptual and Motor Functions. Attachment. Sensorimotor Intelligence and Early Causal Schemes. Understanding the Nature of Objects and Creating Categories. Emotional Development. The Psychosocial Crisis: Trust versus Mistrust. The Central Process for Resolving the Crisis: Mutuality with the Caregiver. The Prime Adaptive Ego Quality and the Core Pathology. Applied Topic: The Role of Parents. Chapter Summary. References. 6. TODDLERHOOD (AGES 2 AND 3). Developmental Tasks. The Psychosocial Crisis: Autonomy versus Shame and Doubt. The Central Process: Imitation. The Prime Adaptive Ego Quality and the Core Pathology. The Impact of Poverty on Psychosocial Development in Toddlerhood. Applied Topic: Day Care. Chapter Summary. References. 7. EARLY SCHOOL AGE (4 TO 6 YEARS). Developmental Tasks. The Psychosocial Crisis: Initiative Versus Guilt. The Central Process: Identification. The Prime Adaptive Ego Quality and the Core Pathology. Applied Topic: School Readiness. Chapter Summary. References. 8. MIDDLE CHILDHOOD (6-12 YEARS). Developmental Tasks. The Psychosocial Crisis: Industry versus Inferiority. The Central Process: Education. The Prime Adaptive Ego Quality and the Core Pathology. Applied Topic: Violence in the Lives of Children. Chapter Summary. References. 9. EARLY ADOLESCENCE (12-18 YEARS). Developmental Tasks. The Psychosocial Crisis: Group Identity versus Alienation. The Central Process: Peer Pressure. The Prime Adaptive Ego Quality and the Core Pathology. Applied Topic: Adolescent Alcohol and Drug Use. Chapter Summary.
---
paper_title: A new view of language acquisition

paper_content:

At the forefront of debates on language are new data demonstrating infants' early acquisition of information about their native language. The data show that infants perceptually “map” critical aspects of ambient language in the first year of life before they can speak. Statistical properties of speech are picked up through exposure to ambient language. Moreover, linguistic experience alters infants' perception of speech, warping perception in the service of language. Infants' strategies are unexpected and unpredicted by historical views. A new theoretical position has emerged, and six postulates of this position are described.
---
paper_title: Maternal Depression and Children’s Antisocial Behavior: Nature and Nurture Effects

paper_content:

Background Children of depressed mothers have elevated conduct problems, presumably because maternal depression disrupts the caregiving environment. Alternatively, the association between maternal depression and children’s antisocial behavior (ASB) may come about because (1) depressed women are likely to have comorbid antisocial personality traits, (2) depressed women are likely to mate and bear children with antisocial men, or (3) children of depressed mothers inherit a genetic liability for psychopathology. Method We used data from the E-Risk Study, a representative British cohort of 1116 twin pairs assessed at 5 and 7 years of age. We tested for environmental mediation of the association between maternal depression during the children’s first 5 years of life and children’s ASB at age 7 years, free from familial liability for ASB. Results Maternal depression occurring after, but not before, the twins’ birth was associated with child ASB and showed a significant dose-response relationship with child ASB at 7 years of age. Parental history of ASPD symptoms accounted for approximately one third of the observed association between maternal depression and children’s ASB, but maternal depression continued to significantly predict children’s ASB. Intraindividual change analyses indicated that children exposed to their mother’s depression between ages 5 and 7 years showed a subsequent increase in ASB by age 7 years. The combination of depression and ASPD symptoms in mothers posed the greatest risk for children’s ASB. Conclusions Studies ignoring genetic transmission overestimate social transmission effects because both genetic and environmental processes are involved in creating risk for ASB in children of depressed mothers. Interventions for depressed mothers aiming to reduce conduct problems in their children should address parents’ antisocial personality, as well as mothers’ depression.
---
paper_title: Cross-language analysis of phonetic units in language addressed to infants.

paper_content:

In the early months of life, infants acquire information about the phonetic properties of their native language simply by listening to adults speak. The acoustic properties of phonetic units in language input to young infants in the United States, Russia, and Sweden were examined. In all three countries, mothers addressing their infants produced acoustically more extreme vowels than they did when addressing adults, resulting in a “stretching” of vowel space. The findings show that language input to infants provides exceptionally well-specified information about the linguistic units that form the building blocks for words.
---
paper_title: A case of hand waving: Action synchrony and person perception

paper_content:

Abstract While previous research has demonstrated that people’s movements can become coordinated during social interaction, little is known about the cognitive consequences of behavioral synchrony. Given intimate links between the systems that regulate perception and action, we hypothesized that the synchronization of movements during a dyadic interaction may prompt increased attention to be directed to an interaction partner, hence facilitate the information that participants glean during a social exchange. Our results supported this prediction. Incidental memories for core aspects of a brief interaction were facilitated following in-phase behavioral synchrony. Specifically, participants demonstrated enhanced memory for an interaction partner’s utterances and facial appearance. These findings underscore the importance of action perception to social cognition.
---
paper_title: Movement coordination in social interaction: Some examples described

paper_content:

Abstract Sound films of social interaction are analysed and some detailed descriptions of the interrelations of the movements of speakers ans listeners are given. These descriptions provide further examples of ‘interactional synchrony’, first described by Condon and Ogston (1966). It is found that the flow of movement in the listener may be rhythmically coordinated with the speech and movements of the speaker. It is also shown how the way in which individuals may be in synchrony with one another can vary, and that these variations are related to their respective roles in the interaction. Some of the implications of the phenomenon of interactional synchrony are discussed.
---
paper_title: Intact automatic imitation of human and robot actions in autism spectrum disorders

paper_content:

The existence of a specialized imitation module in humans is hotly debated. Studies suggesting a specific imitation impairment in individuals with autism spectrum disorders (ASD) support a modular view. However, the voluntary imitation tasks used in these studies (which require socio-cognitive abilities in addition to imitation for successful performance) cannot support claims of a specific impairment. Accordingly, an automatic imitation paradigm (a ‘cleaner’ measure of imitative ability) was used to assess the imitative ability of 16 adults with ASD and 16 non-autistic matched control participants. Participants performed a prespecified hand action in response to observed hand actions performed either by a human or a robotic hand. On compatible trials the stimulus and response actions matched, while on incompatible trials the two actions did not match. Replicating previous findings, the Control group showed an automatic imitation effect: responses on compatible trials were faster than those on incompatible trials. This effect was greater when responses were made to human than to robotic actions (‘animacy bias’). The ASD group also showed an automatic imitation effect and a larger animacy bias than the Control group. We discuss these findings with reference to the literature on imitation in ASD and theories of imitation.
---
paper_title: Movement synchrony and perceived entitativity

paper_content:

Movement synchrony has been theoretically linked to the emergence of a social unit. To empirically investigate whether similar movement rhythms are an antecedent of perceived entitativity, movement rhythms were experimentally manipulated in four studies. Using this novel approach, stick figures waving in synchrony were found to be rated higher on entitativity than stick figures waving in different rhythms (Study 1), and this effect was extended to interactional synchrony, where different movements are performed in the same rhythm (Study 2). Objective differences in movement rhythms are linearly related to ratings of perceived entitativity, and this relationship is partially mediated by the subjectively perceived similarity of movement rhythms (Study 3). These results also held for entitativity judgments for videotaped individuals waving rhythmically (Study 4). These results support the hypothesis that movement rhythms are an important source of information which observers use to infer the extent to which individuals are a social unit.
---
paper_title: Lack of coordination of nonverbal behaviour between patients and interviewers as a potential risk factor to depression recurrence: vulnerability accumulation in depression

paper_content:

Background: Coordination of nonverbal behaviour during interactions is a prerequisite for satisfactory relationships. Lack of coordination may form a risk factor for depression. The 'vulnerability-accumulation hypothesis' assumes that vulnerability to recurrence of depression will increase with increasing experience of depressive episodes. Therefore it is expected that interviewers and patients remitted from a recurrent episode of depression would reach less coordination during a clinical interview compared to interviewers and patients remitted from a first Lifetime episode. Moreover, we assumed that prior severity of depression modifies this reciprocal coordination process. Methods: During discharge interviews, interviewers were videotaped in interaction with remitted patients with unipolar major depression recurrent depression (REC); n = 28; first episode (SEP); n = 22. Durations and frequencies of nonverbal involvement behaviour was registered during the first 15 min. Involvement of the patients consisted of gesticulating, looking at the interviewer, and general head movements; yes nodding and hm, hm, yes, yes reflected involvement of the interviewer. Coordination between patients and interviewers was analyzed per S-min epochs and defined as 'attunement': the absolute difference between patients' and interviewers' involvement. Averaged attunement, its time course and variability (presumably reflecting control of the attunement process) were assessed. Results: The time course of nonverbal attunement differed between the REC and SEP condition. A larger variability of attunement was found in patients remitted from a relatively severely depressed episode, compared to patients remitted from a seven first life time episode. No other significant differences were found. Conclusions: Partial support was found for the notion that nonverbal vulnerability accumulates in depression and that the severity of prior depression modifies this process. (C) 2000 Elsevier Science B.V. All rights reserved.
---
paper_title: The rhythm of rapport: Interpersonal synchrony and social perception

paper_content:

The temporal coordination of behavior during dyadic interactions is a foundation for effective social exchange with synchronized actions enhancing perceptions of rapport and interpersonal connectedness. What has yet to be established, however, are the precise characteristics of behavioral coordination that give rise to such effects. Informed by a dynamical systems approach, the current investigation considered whether judgments of rapport are influenced by the mode of interpersonal synchrony. In two experiments, participants rated the degree of rapport manifest by a simulated pair of walkers exhibiting various configurations of synchronized strides. The results revealed that the highest levels of rapport were associated with the most stable forms of interpersonal coordination (i.e., in-phase and anti-phase synchrony), regardless of whether coordination between the walkers was conveyed via the presentation of visual (Expt. 1a) or auditory (Expt. 1b) cues. These findings underscore the importance of interpersonal coordination to core aspects of social perception.
---
paper_title: H.: Synchrony and cooperation

paper_content:

Armies, churches, organizations, and communities often engage in activities—for example, marching, singing, and dancing—that lead group members to act in synchrony with each other. Anthropologists and sociologists have speculated that rituals involving synchronous activity may produce positive emotions that weaken the psychological boundaries between the self and the group. This article explores whether synchronous activity may serve as a partial solution to the free-rider problem facing groups that need to motivate their members to contribute toward the collective good. Across three experiments, people acting in synchrony with others cooperated more in subsequent group economic exercises, even in situations requiring personal sacrifice. Our results also showed that positive emotions need not be generated for synchrony to foster cooperation. In total, the results suggest that acting in synchrony with others can increase cooperation by strengthening social attachment among group members.
---
paper_title: Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome.

paper_content:

Objective: The authors quantified nonverbal synchrony—the coordination of patient's and therapist's movement—in a random sample of same-sex psychotherapy dyads. The authors contrasted nonverbal synchrony in these dyads with a control condition and assessed its association with session-level and overall psychotherapy outcome. Method: Using an automated objective video analysis algorithm (Motion Energy Analysis; MEA), the authors calculated nonverbal synchrony in (n = 104) videotaped psychotherapy sessions from 70 Caucasian patients (37 women, 33 men, mean age = 36.5 years, SD = 10.2) treated at an outpatient psychotherapy clinic. The sample was randomly drawn from an archive (N = 301) of routinely videotaped psychotherapies. Patients and their therapists assessed session impact with self-report postsession questionnaires. A battery of pre- and postsymptomatology questionnaires measured therapy effectiveness. Results: The authors found that nonverbal synchrony is higher in genuine interactions contrasted with pseudointeractions (a control condition generated by a specifically designed shuffling procedure). Furthermore, nonverbal synchrony is associated with session-level process as well as therapy outcome: It is increased in sessions rated by patients as manifesting high relationship quality and in patients experiencing high self-efficacy. Higher nonverbal synchrony characterized psychotherapies with higher symptom reduction. Conclusions: The results suggest that nonverbal synchrony embodies the patients' self-reported quality of the relationship and further variables of therapy process. This hitherto overlooked facet of therapeutic relationships might prove useful as an indicator of therapy progress and outcome.
---
paper_title: The chameleon effect : The perception-behavior link and social interaction

paper_content:

The chameleon effect refers to nonconscious mimicry of the postures, mannerisms, facial expressions, and other behaviors of one's interaction partners, such that one's behavior passively and unintentionally changes to match that of others in one's current social environment. The authors suggest that the mechanism involved is Has. perception-behavior link, the recently documented finding (e.g., J. A. Bargh, M. Chen, & L. Burrows, 1996) that the mere perception of another's behavior automatically increases the likelihood of engaging in that behavior oneself. Experiment 1 showed that the motor behavior of participants unintentionally matched that of strangers with whom they worked on a task. Experiment 2 had confederates mimic the posture and movements of participants and showed that mimicry facilitates the smoothness of interactions and increases liking between interaction partners. Experiment 3 showed that dispositionally empathic individuals exhibit the chameleon effect to a greater extent than do other people.
---
paper_title: Moving memories: Behavioral synchrony and memory for self and others

paper_content:

Abstract Establishing and maintaining connections with others is central to a fulfilling social life. In this respect, behavioral coordination provides one avenue by which interpersonal linkages can be formed. Drawing from the dynamical systems approach, the present research explored whether temporary interpersonal connections founded on coordinated behavior influence memory for self and others. To do so, we measured participants’ incidental recall of self and other-relevant information after a period of either in-phase or anti-phase interpersonal coordination. While participants in the less stable anti-phase condition demonstrated the typical memory advantage for self-related compared to other-related information, this effect was eliminated when participant and confederate movements displayed in-phase coordination. These results are discussed with respect to the interplay between the systems that support interpersonal synchrony and basic social-cognitive processing.
---
paper_title: Multimodal embodied mimicry in interaction

paper_content:

Nonverbal behavior plays an important role in human-human interaction. One particular kind of nonverbal behavior is mimicry. Behavioral mimicry supports harmonious relationships in social interaction through creating affiliation, rapport, and liking between partners. Affective computing that employs mimicry knowledge and that is able to predict how mimicry affects social situations and relations can find immediate application in human-computer interaction to improve interaction. In this short paper we survey and discuss mimicry issues that are important from that point of view: application in human-computer interaction. We designed experiments to collect mimicry data. Some preliminary analysis of the data is presented.
---
paper_title: What studies of family home movies can teach us about autistic infants: A literature review

paper_content:

Abstract The current study reviewed all prior studies conducted on family home movies of infants who would be later diagnosed with autism spectrum disorder (ASD). Out of 41 original reports found since 1975, we retained 18 studies (317 films, maximum), sorted according to their methodological design using a quality grid. In the first 2 years of life, signs that differentiated children with ASD from children with developmental delays were as follows: less of a response to their name, less looking at others, lower eye contact quality and quantity, less positive facial expression and intersubjective behaviors (e.g., showing shared attention). Studies focusing on regression confirmed the clinical validity of the phenomena. We conclude that findings from home movies studies along with prospective studies have created the bases for identification of infants and toddlers at risk of developing ASD before the 18–24-month period, despite early diagnosis of autism remains a complex challenge.
---
paper_title: Direct and indirect effects of breast milk on the neurobehavioral and cognitive development of premature infants

paper_content:

Eighty-six premature infants were tested to examine the effects of maternal breast milk on infant development. Infants were classified by breast-milk consumption during the hospitalization period (M = 57.4 days) into three groups: those receiving minimal ( 75%) amounts of breast milk. Infants in the three groups were matched for birth weight, gestational age (GA), medical risk, and family demographics. At 37 weeks GA, mother–infant interaction was videotaped, maternal depression self-reported, and neurobehavioral maturation assessed by the Neonatal Behavior Assessment Seale (Brazelton, 1973). At 6 months corrected age, infants were tested with the Bayley II (Bayley, 1993). Infants receiving substantial amounts of breast milk showed better neurobehavioral profiles—in particular, motor maturity and range of state. These infants also were more alert during social interactions, and their mothers provided more affectionate touch. Higher maternal depression scores were associated with lower quantities of breast milk, longer latencies to the first breast-milk feeding, reduced maternal affectionate touch, and lower infant cognitive skills. Maternal affectionate touch moderated the relations between breast milk and cognitive development, with infants receiving a substantial amount of breast milk and frequent touch scoring the highest. In addition to its nutritional value, breast milk may be related to improved maternal mood and interactive behaviors, thereby indirectly contributing to development in premature infants. © 2003 Wiley Periodicals, Inc. Dev Psychobiol 43: 109–119, 2003.
---
paper_title: Parenting Stress, Infant Emotion Regulation, Maternal Sensitivity, and the Cognitive Development of Triplets: A Model for Parent and Child Influences in a Unique Ecology

paper_content:

To examine the development of triplets, 23 sets of triplets were matched with 23 sets of twins and 23 singletons (N 5138). Maternal sensitivity was observed at newborn, 3, 6, and 12 months, and infants’ cognitive and symbolic skills at 1 year. Triplets received lower maternal sensitivity across infancy and exhibited poorer cognitive competencies compared with singletons and twins. The most medically compromised triplet showed the lowest regulation, received lower maternal sensitivity, and demonstrated the weakest outcomes compared with siblings. Structural modeling charted three levels of influence on cognitive outcomes: direct, indirect, and contextual. The triplet ecology provides a context for assessing the relations among infant inborn dispositions, the rearing environment, and the role of exclusive parenting in development. Although the absolute numbers of triplets and higher order multiples born yearly may be relatively small, triplets are the most rapidly growing birth population. Rates of triplet births in the United States have increased by tenfold since the 1980s, and the same growth rates are reported in other industrial countries, such as England, France, Belgium, Hol
---
paper_title: Inferring truth from multiple annotators for social interaction analysis

paper_content:

This study focuses on incorporating knowledge from multiple annotators into a machine-learning framework for detecting psychological traits using multimodal data. We present a model that is designed to exploit the judgements of multiple annotators on a social trait labeling task. Our two-stage model first estimates a ground truth by modeling the annotators using both the annotations and annotators’ self-reported confidences. In the second stage, we train a classifier using the estimated ground truth as labels. Our experiments on a dominance estimation task in a group interaction scenario on the DOME corpus, in addition to synthetically generated data, give satisfactory results, outperforming the commonly used majority voting as well as other approaches in the literature.
---
paper_title: Spatiotemporal Coding in ANVIL

paper_content:

We present a new coding mechanism, spatiotemporal coding, that allows coders to annotate points and regions in the video frame by drawing directly on the screen. Coders can not only attach labels to time intervals in the video but can specify a possibly moving region on the video screen. This opens up the spatial dimension for multi-track video coding and is an essential asset in almost every area of video coding, e.g. gesture coding, facial expression coding, encoding semantics for information retrieval etc. We discuss conceptual variants, design decisions and the relation to the MPEG-7 standard and tools.
---
paper_title: Skin-to-skin contact (kangaroo care) promotes self-regulation in premature infants: Sleep-wake cyclicity, arousal modulation, and sustained exploration.

paper_content:

The effect of mother–infant skin-to-skin contact (kangaroo care, or KC) on self-regulatory processes of premature infants was studied. Seventy-three infants who received KC were compared with 73 infants matched for birth weight, gestational age, medical risk, and family demographics. State organization was measured in 10-s epochs over 4 hr before KC and again at term. No differences between KC infants and controls were found before KC. At term, KC infants showed more mature state distribution and more organized sleep–wake cyclicity. At 3 months, KC infants had higher thresholds to negative emotionality and more efficient arousal modulation while attending to increasingly complex stimuli. At 6 months, longer duration of and shorter latencies to mother–infant shared attention and infant sustained exploration in a toy session were found for KC infants. The results underscore the importance of maternal body contact for infants’ physiological, emotional, and cognitive regulatory capacities. Premature birth deprives the infant not only of the protective and nurturing environment of the mother’s womb but also of the benefits of maternal proximity and contact. Prematurity thus creates a unique phase in the life of the human infant, a period in which portions of the newborn’s prenatal development take place in an extrauterine environment under conditions of maternal separation. Although the effects of prematurity on the physiology and behavior of the developing child have been frequently studied, far less research has examined the associations between intervention applied in the neonatal period and developmental consequences. Most studies on the long-term effects of intervention for premature infants have shown either general gains in infants’ cognitive abilities or a global increase in maternal sensitivity (Als et al., 1994; Bernard & Bee, 1983; Resnick, Eyler, Nelson, Eitzman, & Bucciarelli, 1987; Scarr-Salapatek & Williams, 1973; Solkoff, Yaffe, Weintraub, & Blase, 1969). Such global findings, however, provide limited understanding of the mechanisms that link intervention to specific outcomes. This study examined the effects of mother–infant skin-to-skin contact in the neonatal period on the development of selfregulatory skills in premature infants. Specifically, we examined the effects of such contact on the infant’s capacities to regulate sleep and wake states, organize behavior, regulate negative emotions, modulate arousal, coordinate attention to mother and an object, and sustain effortful exploration of the environment. Skinto-skin contact was expected to modify each of the adverse conditions associated with the prematurity state, that is, maternal separation, stimulus overload caused by routine nursery care, sensory deprivation resulting from limited tactile and maternal contact during the incubation period, and pain exposure (Als et al., 1994; Feldman & Eidelman, 1998; White-Traut & Nelson, 1988). We hypothesized that skin-to-skin contact would promote physiological and behavior organization as expressed in more mature state organization in the neonatal period and in improved emotion regulation, joint engagement, and exploratory skills in the first
---
paper_title: Mother–Child Touch Patterns in Infant Feeding Disorders: Relation to Maternal, Child, and Environmental Factors

paper_content:

ABSTRACT Objective To examine mother and child's touch patterns in infant feeding disorders within a transactional framework. Method Infants (aged 9–34 months) referred to a community-based clinic were diagnosed with feeding disorders ( n = 20) or other primary disorder ( n = 27) and were case matched with nonreferred controls ( n = 47). Mother–child play and feeding were observed and the home environment was assessed. Microcoding detected touch patterns, response to partner's touch, and proximity at play. Relational behaviors were coded during feeding. Results Compared with infants with other primary disorder and case-matched controls, less maternal affectionate, proprioceptive, and unintentional touch was observed in those with feeding disorders. Children with feeding disorders displayed less affectionate touch, more negative touch, and more rejection of the mother's touch. More practical and rejecting maternal responses to the child's touch were observed, and children were positioned more often out of reach of the mothers' arms. Children with feeding disorders exhibited more withdrawal during feeding and the home environment was less optimal. Feeding efficacy was predicted by mother–child touch, reduced maternal depression and intrusiveness, easy infant temperament, and less child withdrawal, controlling for group membership. Conclusions Proximity and touch are especially disturbed in feeding disorders, suggesting fundamental relationship difficulties. Mothers provide less touch that supports growth, and children demonstrate signs of touch aversion. Touch patterns may serve as risk indicators of potential growth failure.
---
paper_title: Disorganized rhythm and synchrony: Early signs of autism and Rett syndrome

paper_content:

We interpret early age-related developments in intentions and socially responsive behaviour with data from home videos of infants who later develop autism or Rett syndrome. Detailed evidence is given from a micro-analytic study of videos of monozygotic twin girls at 11 months, one of whom became autistic in the second year. Changes in this twin’s attention, motor tonus, initiative and emotion reduce her prospective control of movements and her anticipations in awareness compared to her sister. These changes were reflected in the child’s asynchronous social behaviour, which frustrated the father’s attempts to support her attempts to walk, share toys, or play a game, confusing his anticipations, and this further reduced mutual attention and joint activity. Observations of the development of girls with Rett syndrome in the first year reveal changes in motor coordination, attention and communicative initiative, indicative of a failure of intrinsic core brain regulations of neural development and conscious activity. Notwithstanding that the two conditions show clear differences in both brain growth and early development of skills and sociability, the first signs of autism and Rett syndrome have important similarities. We conclude with recommendations for an approach to early diagnosis and treatment, applicable for the whole range of developmental brain disorders, including Rett syndrome and autism, that attempts to identify residual capacities for sympathetic motivation and collaborative learning—an approach that deliberately tries to support weakened rhythmic impulses for motor, perceptual and communicative functions in the growing infant brain. q 2005 Elsevier B.V. All rights reserved.
---
paper_title: Parent–infant synchrony and the construction of shared timing; physiological precursors, developmental outcomes, and risk conditions

paper_content:

Synchrony, a construct used across multiple fields to denote the temporal relationship between events, is applied to the study of parent-infant interactions and suggested as a model for intersubjectivity. Three types of timed relationships between the parent and child's affective behavior are assessed: concurrent, sequential, and organized in an ongoing patterned format, and the development of each is charted across the first year. Viewed as a formative experience for the maturation of the social brain, synchrony impacts the development of self-regulation, symbol use, and empathy across childhood and adolescence. Different patterns of synchrony with mother, father, and the family and across cultures describe relationship-specific modes of coordination. The capacity to engage in temporally-matched interactions is based on physiological mechanisms, in particular oscillator systems, such as the biological clock and cardiac pacemaker, and attachment-related hormones, such as oxytocin. Specific patterns of synchrony are described in a range of child-, parent- and context-related risk conditions, pointing to its ecological relevance and usefulness for the study of developmental psychopathology. A perspective that underscores the organization of discrete relational behaviors into emergent patterns and considers time a central parameter of emotion and communication systems may be useful to the study of interpersonal intimacy and its potential for personal transformation across the lifespan.
---
paper_title: Neonate Movement Is Synchronized with Adult Speech: Interactional Participation and Language Acquisition

paper_content:

As early as the first day of life, the human neonate moves in precise and sustained segments of movement that are synchronous with the articulated structure of adult speech. These observations suggest a view of development of the infant as a participant at the outset in multiple forms of interactional organization, rather than as an isolate.
---
paper_title: Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome.

paper_content:

Objective: The authors quantified nonverbal synchrony—the coordination of patient's and therapist's movement—in a random sample of same-sex psychotherapy dyads. The authors contrasted nonverbal synchrony in these dyads with a control condition and assessed its association with session-level and overall psychotherapy outcome. Method: Using an automated objective video analysis algorithm (Motion Energy Analysis; MEA), the authors calculated nonverbal synchrony in (n = 104) videotaped psychotherapy sessions from 70 Caucasian patients (37 women, 33 men, mean age = 36.5 years, SD = 10.2) treated at an outpatient psychotherapy clinic. The sample was randomly drawn from an archive (N = 301) of routinely videotaped psychotherapies. Patients and their therapists assessed session impact with self-report postsession questionnaires. A battery of pre- and postsymptomatology questionnaires measured therapy effectiveness. Results: The authors found that nonverbal synchrony is higher in genuine interactions contrasted with pseudointeractions (a control condition generated by a specifically designed shuffling procedure). Furthermore, nonverbal synchrony is associated with session-level process as well as therapy outcome: It is increased in sessions rated by patients as manifesting high relationship quality and in patients experiencing high self-efficacy. Higher nonverbal synchrony characterized psychotherapies with higher symptom reduction. Conclusions: The results suggest that nonverbal synchrony embodies the patients' self-reported quality of the relationship and further variables of therapy process. This hitherto overlooked facet of therapeutic relationships might prove useful as an indicator of therapy progress and outcome.
---
paper_title: Maternal representations, infant psychiatric status, and mother–child relationship in clinic-referred and non-referred infants

paper_content:

To examine the relations between maternal representations, infant socio-emotional difficulties, and mother–child relational behavior, 49 clinic-referred infants and their mothers were compared to 30 non-referred controls. Clinic-referred infants’ psychiatric status was determined with the DC 0-3-R classification of Zeanah and Benoit (Child Adolesc Psychiatry Clin N Am 4:539–554, 1995) and controls were screened for socio-emotional difficulties. Mothers were interviewed with the parent development interview (Aber et al. in The parent development interview. Unpublished manuscript, 1985) and dyads were observed in free play and problem-solving interactions. Group differences emerged for maternal representations and relational behaviors. Representations of clinic-referred mothers were characterized by lower joy, coherence, and richness, and higher anger experienced in the mother–infant relationship compared to controls. During free play, clinic-referred mothers showed lower sensitivity and higher intrusiveness and provided less adequate instrumental and emotional assistance and support during problem solving. Referred children showed lower social engagement during free play. Associations were found among maternal representations, maternal interactive behavior, child social engagement, and the child’s ability to self-regulate during a challenging task. These findings provide empirical support for theoretical and clinical perspectives suggesting a reciprocal link between maternal negative representations and mother and child’s maladaptive behaviors in the context of early socio-emotional difficulties and mental health referrals.
---
paper_title: The Mother–Infant Feeding Relationship Across the First Year and the Development of Feeding Difficulties in Low‐Risk Premature Infants

paper_content:

Although feeding problems are common during infancy and are typically accompanied by relational difficulties, little research observed the mother–infant feeding relationship across the first year as an antecedent to the development of feeding difficulties. We followed 76 low-risk premature infants and their mothers from the transition to oral feeding in the neonatal period to the end of the first year. Prior to hospital discharge, microlevel patterns of maternal touch and gaze were coded during feeding and nonfeeding interactions, global patterns of maternal adaptation were assessed, and infants' neurobehavioral status was tested. Psychomotor development was evaluated at 4 months. At 1 year, feeding difficulties were determined on the basis of maternal interview and direct observations of feeding interactions. Mothers of infants who exhibited feeding difficulties at 1 year showed less affectionate touch and gaze during nonfeeding interactions and more gaze aversion and lower adaptability during feeding in...
---
paper_title: Behavioral and judged coordination in adult informal social interactions: Vocal and kinesic indicators.

paper_content:

Coordination in social interaction means that persons adjust their actions to those of their partners. Common methods for measuring coordination include judgments and behavioral covariation. Sixteen 1-min segments of interaction were chosen (8 high and 8 low in behavioral coordination). In Study 1, 51 people judged the 16 segments, rating each for coordination. Study 2 (N = 17) used different items. Study 3 (N = 22) replicated Study 2 without sound and with a mosaic pattern imposed on the faces. Results indicated judges were reliable, able to distinguish high from low coordination interactions on the basis of 1-min slices for male but not female dyads. Segments judged to be coordinated had partners smiling in synchrony but with complementary patterns of gazing and gesturing. Both measures correlated with conversational satisfaction, but only behavioral coordination predicted attraction.
---
paper_title: Entrainment in Speech Preceding Backchannels.

paper_content:

In conversation, when speech is followed by a backchannel, evidence of continued engagement by one's dialogue partner, that speech displays a combination of cues that appear to signal to one's interlocutor that a backchannel is appropriate. We term these cues back-channel-preceding cues (BPC)s, and examine the Columbia Games Corpus for evidence of entrainment on such cues. Entrainment, the phenomenon of dialogue partners becoming more similar to each other, is widely believed to be crucial to conversation quality and success. Our results show that speaking partners entrain on BPCs; that is, they tend to use similar sets of BPCs; this similarity increases over the course of a dialogue; and this similarity is associated with measures of dialogue coordination and task success.
---
paper_title: Achieving Rapport with Turn-by-Turn, User-Responsive Emotional Coloring

paper_content:

People in dialog use a rich set of nonverbal behaviors, including variations in the prosody of their utterances. Such behaviors, often emotion-related, call for appropriate responses, but today's spoken dialog systems lack the ability to do this. Recent work has shown how to recognize user emotions from prosody and how to express system-side emotions with prosody, but demonstrations of how to combine these functions to improve the user experience have been lacking. Working with a corpus of conversations with students about graduate school, we analyzed the emotional states of the interlocutors, utterance-by-utterance, using three dimensions: activation, evaluation, and power. We found that the emotional coloring of the speaker's utterance could be largely predicted from the emotion shown by her interlocutor in the immediately previous utterance. This finding enabled us to build Gracie, the first spoken dialog system that recognizes a user's emotional state from his or her speech and gives a response with appropriate emotional coloring. Evaluation with 36 subjects showed that they felt significantly more rapport with Gracie than with either of two controls. This shows that dialog systems can tap into this important level of interpersonal interaction using today's technology.
---
paper_title: Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series

paper_content:

Cross-correlation and most other longitudinal analyses assume that the association between 2 variables is stationary. Thus, a sample of occasions of measurement is expected to be representative of the association between variables regardless of the time of onset or number of occasions in the sample. The authors propose a method to analyze the association between 2 variables when the assumption of stationarity may not be warranted. The method results in estimates of both the strength of peak association and the time lag when the peak association occurred for a range of starting values of elapsed time from the beginning of an experiment.
---
paper_title: Frequency detuning of the phase entrainment dynamics of visually coupled rhythmic movements

paper_content:

An order parameter equation for correlated limb movements was applied to rhythmic coordination between the limbs of two people. The interlimb coordination was established and maintained through vision. Manipulations of frequency competition, coupled frequency, and intended mode (in-phase or anti-phase) produced equilibria and fluctuations in relative phase predicted by the order parameter equation and confirmed originally in within-person coordination. It was concluded that there is an elementary coordination dynamics governing the rhythmic coordination between organisms as well as between components of a single organism.
---
paper_title: Towards visual and vocal mimicry recognition in human-human interactions

paper_content:

During face-to-face interpersonal interaction, people have a tendency to mimic each other. People not only mimic postures, mannerisms, moods or emotions, but they also mimic several speech-related behaviors. In this paper we describe how visual and vocal behavioral information expressed between two interlocutors can be used to detect and identify visual and vocal mimicry. We investigate expressions of mimicry and aim to learn more about in which situation and to what extent mimicry occurs. The observable effects of mimicry can be explored by representing and recognizing mimicry using visual and vocal features. In order to automatically analyze how to extract and integrate this behavioral information into a multimodal mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration in both audio and video modalities.
---
paper_title: An Audio-Visual Approach to Measuring Discourse Synchrony in Multimodal Conversation Data

paper_content:

This paper describes recent work on the automatic extraction of visual and audio parameters relating to the detection of synchrony in discourse, and to the modelling of active listening for advanced speech technology. It reports findings based on image processing that reliably identify the strong entrainment between members of a group conversation, and describes techniques for the extraction and analysis of such information.
---
paper_title: Measuring entrainment in small groups of musicians

paper_content:

This research is concerned with synchronization (or entrainment), and in particular on emotional synchronization, which is likely to be an important driver of collaborative processes. We focus on ensemble musical performance, an ideal test-bed for the development of models and techniques for measuring creative social interaction in an ecologically valid framework. Ongoing work and early results on the automated analysis in real-time of non-verbal cues related to expressive gesture in social interaction is briefly presented.
---
paper_title: Prosodic and gestural expression of interactional agreement

paper_content:

Conversational interactions are cooperatively constructed activities in which participants negotiate their entrances, turns and alignments with other speakers, oftentimes with an underlying long-term objective of obtaining some agreement. Obtaining a final and morally binding accord in a conversational interaction is of importance in a great variety of contexts, particularly in psychotherapeutic interactions, in contractual negotiations or in educational contexts. Various prosodic and gestural elements in a conversational interaction can be interpreted as signals of a speaker's agreement and they are probably of importance in the emergence of an accord in a conversational exchange. In this paper, we survey the social and psychological context of agreement seeking, as well as the existing literature on the visual and prosodic measurement of agreement in conversational settings.
---
paper_title: Synchrony - a core concept for a constructivist approach to psychotherapy

paper_content:

Synchrony is a pervasive concept relevant to diverse domains in physics, biology, and the social sciences. This article reviews some of the evidence both from natural and human systems. Our main focus is on the role of synchrony in psychotherapy research. Its association with empathy, rapport and the therapeutic relationship has been pointed out repeatedly, yet close evaluation of empirical studies suggests that the evidence remains inconclusive. Particularly in naturalistic studies, research employing synchrony as a specific measure is still lacking. We suggest a new approach to studying synchrony in psychotherapies under field conditions: Image-differencing is a video-based algorithm quantifying the amount of movement in pre-selected regions. Our methodology can be compared to the approach from the seminal work of Condon & Ogston (1966). The statistical analysis we employ detects synchrony on a global level, irrespective of the specific body parts moving. Synchrony thus defined is considered a general measure of coordination of interacting individuals. Exemplary data show the feasibility of this approach and its potential contribution to psychotherapy research.
---
paper_title: Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension

paper_content:

Looking To Understand: The Coupling Between Speakers’ and Listeners’ Eye Movements and its Relationship to Discourse Comprehension Daniel C. Richardson (richardson@psych.stanford.edu) Department of Psychology, Stanford University Stanford, CA 94305, USA Rick Dale (rad28@cornell.edu) Department of Psychology, Cornell University Ithaca, NY 14853, USA Abstract discussing a diagram drawn on a whiteboard, figuring out together how to do something on a computer, or talking during a movie. Uniquely poised between perception and cognition, eye movements can reveal cognitive processes such as speech planning, language comprehension, memory, mental imagery and decision making. The current experiment investigates whether the eye movements of a speaker and a listener to a visual common ground can provide insight into a discourse. While their eye movements were being recorded, participants spoke extemporaneously about a TV show whose cast members they were viewing. Later, other participants listened to these speeches while their eyes were tracked. Within this naturalistic paradigm using spontaneous speech, a number of results linking eye movements to speech comprehension, speech production and memory were replicated. More importantly, a cross- recurrence analysis demonstrated that speaker and listener eye movements were coupled, and that the strength of this relationship positively correlated with listeners’ comprehension. Just as the mental state of a single person can be reflected in patterns of eye movements, the commonality of mental states that is brought about by successful communication is mirrored in a similarity between speaker and listener’s eye movements. Eye movement Research Introduction Imagine standing in front of a painting, discussing it with a friend. As you talk, your eyes will scan across the image, moving approximately three times a second. They will be drawn by characteristics of the image itself, areas of contrast or detail, as well as features of the objects or people portrayed. Eye movements are driven both by properties of the visual world and processes in a person’s mind. Your gaze might also be influenced by what your friend is saying, what you say in reply, what is thought but not said, and where you agree and disagree. If this is so, what is the relationship between your eye movements and those of your friend? How is that relationship related to the flow of conversation between you? Language use often occurs within rich visual contexts such as this, and the interplay between linguistic processes and visual perception is of increasing interest to psycholinguists and vision researchers (Henderson & Ferreira, 2004). As yet, however, such processes have been limited to experiments that examine the eye movements of the speaker or the listener in isolation. Language use, more often than not, occurs within a richer social context as well. Direct eye contact between conversants plays an interesting, crucial role in coordinating a conversation (Bavelas, Coates, & Johnson, 2002), and in conveying various attitudes or social roles (Argyle & Cook, 1976). The focus of the current experiment, however, is cases such as those introduced at the outset, where conversants are not looking at each other, but at some visual scene that is the topic of the conversation. More common examples might be Eye movements of a speaker If a speaker is asked to describe a simple scene, they will fixate the objects in the order in which they are mentioned, around 900ms before naming them (Griffin & Bock, 2000; Meyer, Sleiderink, & Levelt, 1998). Since such pictures can be identified rapidly, it is argued that during this time speakers are not just retrieving words but selecting and planning which to use. Eye movements of a listener Eye movement research has shown that there is a tight interdependence between speech recognition and visual perception. Eye movements to potential referents for a word can provide evidence for a lexical item being recognized before the word is finished being spoken. The link between visual and linguistic processing can also be seen in eye movements that disambiguate syntactic structures (Tanenhaus, Spivey Knowlton, Eberhard, & Sedivy, 1995) and anticipate the future agents of actions (Kamide, Altmann, & Haywood, 2003). Recent studies of the eye- movements of a participant engaged in a conversation with another naive participant reveal a remarkable sensitivity to the referential domains established by the task, the visual context and the preceding conversation (Brown-Schmidt, Campana, & Tanenhaus, 2004). Qualitatively, eye movement research reveals a very close, time-locked integration between visual and linguistic processing (Tanenhaus, Magnuson, Dahan, & Chambers, 2000). Although fixation times are heavily modulated by context, as a very rough quantitative guide, research suggests that listeners will fixate an object around 400-800ms after the name onset.
---
paper_title: A System for Real-Time Multimodal Analysis of Nonverbal Affective Social Interaction in User-Centric Media

paper_content:

This paper presents a multimodal system for real-time analysis of nonverbal affective social interaction in small groups of users. The focus is on two major aspects of affective social interaction: the synchronization of the affective behavior within a small group and the emergence of functional roles, such as leadership. A small group of users is modeled as a complex system consisting of single interacting components that can auto-organize and show global properties. Techniques are developed for computing quantitative measures of both synchronization and leadership. Music is selected as experimental test-bed since it is a clear example of interactive and social activity, where affective nonverbal communication plays a fundamental role. The system has been implemented as software modules for the EyesWeb XMI platform (http://www.eyesweb.org). It has been used in experimental frameworks (a violin duo and a string quartet) and in real-world applications (in user-centric applications for active music listening). Further application scenarios include entertainment, edutainment, therapy and rehabilitation, cultural heritage, and museum applications. Research has been carried out in the framework of the EU-ICT FP7 Project SAME (http://www.sameproject.eu).
---
paper_title: Toward a Real-Time Automated Measure of Empathy and Dominance

paper_content:

This paper presents a number of algorithms andrelated software modules for the automated analysisin real-time of non-verbal cues related to expressivegesture in social interaction. Analysis is based onPhase Synchronisation (PS) and RecurrenceQuantification Analysis. We start from the hypothesisthat PS is one of the low-level social signalsexplaining empathy and dominance in a small groupof users. Real- time implementation of the algorithmsis available in the EyesWeb XMI Social SignalProcessing Library. Applications of the approach isin user-centric media and in particular in activemusic listening (EU ICT FP7 Project SAME).
---
paper_title: Effects of Social and Physical Variables on Between-Person Visual Coordination

paper_content:

Between-person visual coordination of rhythmic limb movements was investigated under manipulation of both social and physical variables. Each member of a pair of participants oscillated a hand-held pendulum at the same tempo, with both visually coordinating their movements so as to maintain the alternate phase mode. Participants' social competence was assessed and they were paired to produce three types of combinations: high-high (HH), low-low (LL), and high-low (HL). The frequency of oscillation (set by a metronome) and the preferred frequency differ- ence of the pendulum combinations (set by changing the relative lengths of the pendulums to be coordinated) were manipulated. Mean relative phase angle 4 and its variability were measured as indices of coordination. Standard results from interlimb coordination were replicated in the manipulation of the physical vari- ables: (a) More breakdowns in phase locking (defined as mean 4s outside the stable 130 deg. to 230 deg. range) were observed for the higher fr...
---
paper_title: Phase transitions and critical fluctuations in the visual coordination of rhythmic movements between people

paper_content:

By watching each other's lower oscillating leg, 2 seated Ss kept a common tempo and a particular phase relation of either 0 degrees (symmetric mode) or 180 degrees (alternate mode). This study investigated the differential stability of the 2 phase modes. In Experiment 1, in which Ss were instructed to remain in the initial phase mode, the alternate phase mode was found to be less stable as the frequency of oscillation increased. In addition, analysis of the nonsteady state cycles revealed evidence of a switching to the symmetric phase mode for the initial alternate phase mode trials. In Experiments 2 and 3, Ss were instructed to remain at a noninitial phase angle if it was found to be more comfortable. The transition observed between the 2 phase modes satisfies the criteria of a physical bifurcation--hysteresis, critical fluctuations, and divergence--and is consonant with previous findings on transitions in limb coordination within a person. Language: en
---
paper_title: Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome.

paper_content:

Objective: The authors quantified nonverbal synchrony—the coordination of patient's and therapist's movement—in a random sample of same-sex psychotherapy dyads. The authors contrasted nonverbal synchrony in these dyads with a control condition and assessed its association with session-level and overall psychotherapy outcome. Method: Using an automated objective video analysis algorithm (Motion Energy Analysis; MEA), the authors calculated nonverbal synchrony in (n = 104) videotaped psychotherapy sessions from 70 Caucasian patients (37 women, 33 men, mean age = 36.5 years, SD = 10.2) treated at an outpatient psychotherapy clinic. The sample was randomly drawn from an archive (N = 301) of routinely videotaped psychotherapies. Patients and their therapists assessed session impact with self-report postsession questionnaires. A battery of pre- and postsymptomatology questionnaires measured therapy effectiveness. Results: The authors found that nonverbal synchrony is higher in genuine interactions contrasted with pseudointeractions (a control condition generated by a specifically designed shuffling procedure). Furthermore, nonverbal synchrony is associated with session-level process as well as therapy outcome: It is increased in sessions rated by patients as manifesting high relationship quality and in patients experiencing high self-efficacy. Higher nonverbal synchrony characterized psychotherapies with higher symptom reduction. Conclusions: The results suggest that nonverbal synchrony embodies the patients' self-reported quality of the relationship and further variables of therapy process. This hitherto overlooked facet of therapeutic relationships might prove useful as an indicator of therapy progress and outcome.
---
paper_title: Emotional entrainment in music performance

paper_content:

This work aims at defining a computational model of human emotional entrainment. Music, as a non-verbal language to express emotions, is chosen as an ideal test bed for these aims. We start from multimodal gesture and motion signals, recorded in a real world collaborative condition in an ecological setting. Four violin players were asked to play, alone or in duo, a music fragment in two different perceptual feedback modalities and in four different emotional states. We focused our attention on phase synchronisation of the head motions of the players. From observation by subjects (musicians and observers), an evidence of entrainment emerges between players. The preliminary results, based on a reduced data set, however do not grasp fully this phenomenon. A more extended analysis is current subject of investigation.
---
paper_title: Social coordination dynamics: Measuring human bonding

paper_content:

Abstract Spontaneous social coordination has been extensively described in natural settings but so far no controlled methodological approaches have been employed that systematically advance investigations into the possible self-organized nature of bond formation and dissolution between humans. We hypothesized that, under certain contexts, spontaneous synchrony—a well-described phenomenon in biological and physical settings—could emerge spontaneously between humans as a result of information exchange. Here, a new way to quantify interpersonal interactions in real time is proposed. In a simple experimental paradigm, pairs of participants facing each other were required to actively produce actions, while provided (or not) with the vision of similar actions being performed by someone else. New indices of interpersonal coordination, inspired by the theoretical framework of coordination dynamics (based on relative phase and frequency overlap between movements of individuals forming a pair) were developed and us...
---
paper_title: Evaluating the Dynamics of Unintended Interpersonal Coordination

paper_content:

Past research has shown that interpersonal interactions are characterized by a tacit coordination of motor movements of the participants and has suggested that the emergent synchrony might be explained by a coupled oscillator dynamic. This study investigates whether unintended between-person coordination can be demonstrated in a laboratory task that will allow an evaluation of whether such dynamical processes are involved. Ten pairs of participants performed a simple rhythmic task in which they had visual information about each other's movements but had no goal to coordinate. A cross-spectral analysis of the movements revealed higher coherence and a distribution of relative phase angles that was dominated by values near 0 deg. and 180 deg. These results support the hypothesis that dynamical organizing principles are involved in natural interpersonal synchrony.
---
paper_title: Spatiotemporal Symmetry and Multifractal Structure of Head Movements during Dyadic Conversation.

paper_content:

This study examined the influence of sex, social dominance, and context on motion-tracked head movements during dyadic conversations. Windowed cross-correlation analyses found high peak correlation between conversants' head movements over short (≈2-s) intervals and a high degree of nonstationarity. Nonstationarity in head movements was found to be positively related to the number of men in a conversation. Surrogate data analysis offsetting the conversants' time series by a large lag was unable to reject the null hypothesis that the observed high peak correlations were unrelated to short-term coordination between conversants. One way that high peak correlations could be observed when 2 time series are offset by a large time lag is for each time series to exhibit self-similarity over a range of scales. Multifractal analysis found small-scale fluctuations to be persistent, τ(q) 0.5. These results are consistent with a view that symmetry is formed between conversants over short intervals and that this symmetry is broken at longer, irregular intervals.
---
paper_title: Body Movement Synchrony in Psychotherapeutic Counseling: A Study Using the Video-Based Quantification Method

paper_content:

Body movement synchrony (i.e. rhythmic synchronization between the body movements of interacting partners) has been described by subjective impressions of skilled counselors and has been considered to reflect the depth of the client-counselor relationship. This study analyzed temporal changes in body movement synchrony through a video analysis of client-counselor dialogues in counseling sessions. Four 50-minute psychotherapeutic counseling sessions were analyzed, including two negatively evaluated sessions (low evaluation groups) and two positively evaluated sessions (high evaluation groups). In addition, two 50-minute ordinary advice sessions between two high school teachers and the clients in the high rating group were analyzed. All sessions represent role-playing. The intensity of the participants' body movement was measured using a video-based system. Temporal change of body movement synchrony was analyzed using moving correlations of the intensity between the two time series. The results revealed (1) A consistent temporal pattern among the four counseling cases, though the moving correlation coefficients were higher for the high evaluation group than the low evaluation group and (2) Different temporal patterns for the counseling and advice sessions even when the clients were the same. These results were discussed from the perspective of the quality of client-counselor relationship.
---
paper_title: Mutual interpersonal postural constraints are involved in cooperative conversation.

paper_content:

The research was designed to evaluate interpersonal coordination during conversation with a new measurement tool. The experiment uses an analysis based on recurrence strategies, known as cross recurrence quantification, to evaluate the shared activity between 2 postural time series in reconstructed phase space. Pairs of participants were found to share more locations in phase space (greater recurrence) in conditions where they were conversing with one another to solve a puzzle task than in conditions in which they conversed with others. The trajectories of pairs of participants also showed less divergence when they conversed with each other than when they conversed with others as well. This is offered as objective evidence of interpersonal coordination of postural sway in the context of a cooperative verbal task.
---
paper_title: Multimodal coordination: exploring relevant features and measures

paper_content:

The decisive role of interpersonal coordination in social interactions is well established. As human beings, we are "experts" in decoding and producing social signals, learning since our birth. Though, equipping a machine with the same abilities represents a great challenge to design socially-accepted interfaces. Assessing automatically this coordination requires selecting the most relevant features. In this paper, various audio and visual features are investigated to characterize interactional synchrony in a cooperative task. A hierarchical representation of the similarity between extracted features is proposed. The properties of time-domain (correlation coefficients) and frequency-domain (coherence) synchrony measures are confronted. The presented results support the evidence that synchrony is a multimodal phenomenon. Moreover, it can be assessed both on local (short-length windows of interaction) or global scales (entire interaction).
---
paper_title: A Comparison of Intra- and Interpersonal Interlimb Coordination: Coordination Breakdowns and Coupling Strength

paper_content:

Intraand interpersonal inter]imb coordination of pendulums swung from the wrist was investigated. For both kinds of coordination, the steady state and breakdown of bimanual rhythmic coordination as indexed by the time series of the relative phase angle it were studied under the manipulation of coordination mode, frequency of oscillation, and the difference in the eigenfrequencies (preferred tempos) of the individual oscillating limbs. The properties observed for both intraand interpersonal coordination were those predicted by a dynamical model of rhythmic coordination that considers the coordinated limbs coupled to be nonlinear oscillators. Using a regression method, the coupling strengths of the coupled system were recovered. As predicted by the dynamical model, the strength of the dynamic was generally greater for the in-phase than the anti-phase mode and decreased with increasing frequency. Further, the strength of the interpersonal interlimb coupling was weaker than that of intrapersonal interlimb coupling.
---
paper_title: Symmetry building and symmetry breaking in synchronized movement

paper_content:

Mirror symmetry between individuals is a behavioral phenomenon that is commonly observed in conversation: individuals tend to mimic each other’s postures and gestures as a part of a shared dialog. The present work studies the process of symmetry building and symmetry breaking in the movements of pairs of individuals while imitating each others’ movements in dance. Spatial and temporal symmetries are found in the overall velocities from the results of full body motion tracking. This work was supported in part by NIA grant R29 AG14983-01 and a grant from the University of Notre Dame. We wish to thank Elizabeth Marsh who assisted in gathering these data. Corresponding author: Steven M. Boker, Department of Psychology, The University of Notre Dame, Notre Dame Indiana 46556, USA; sboker@nd.edu; http://www.nd.edu/ sboker. SYMMETRY IN SYNCHRONIZED MOVEMENT 2
---
paper_title: Measuring Multimodal Synchrony for Human-Computer Interaction

paper_content:

Nonverbal synchrony is an important and natural element in human-human interaction. It can also play various roles in human-computer interaction. In particular this is the case in the interaction between humans and the virtual humans that inhabit our cyber worlds. Virtual humans need to adapt their behavior to the behavior of their human interaction partners in order to maintain a natural and continuous interaction synchronization. This paper surveys approaches to modeling synchronization and applications where this modeling is important. Apart from presenting this framework, we also present a quantitative method for measuring the level of nonverbal synchrony in an interaction and observations on future research that allows embedding such methods in models of interaction behavior of virtual humans.
---
paper_title: Differences in head orientation behavior for speakers and listeners: An experiment in a virtual environment

paper_content:

An experiment was conducted to investigate whether human observers use knowledge of the differences in focus of attention in multiparty interaction to identify the speaker amongst the meeting participants. A virtual environment was used to have good stimulus control. Head orientations were displayed as the only cue for focus attention. The orientations were derived from a corpus of tracked head movements. We present some properties of the relation between head orientations and speaker--listener status, as found in the corpus. With respect to the experiment, it appears that people use knowledge of the patterns in focus of attention to distinguish the speaker from the listeners. However, the human speaker identification results were rather low. Head orientations (or focus of attention) alone do not provide a sufficient cue for reliable identification of the speaker in a multiparty setting.
---
paper_title: Nonverbal synchrony or random coincidence? how to tell the difference

paper_content:

Nonverbal synchrony in face-to-face interaction has been studied in numerous empirical investigations focusing on various communication channels. Furthermore, the pervasiveness of synchrony in physics, chemistry and biology adds to its face-validity. This paper is focused on establishing criteria for a statistical evaluation of synchrony in human interaction. When assessing synchrony in any communication context, it is necessary to distinguish genuine synchrony from pseudosynchrony, which may arise due to random coincidence. By using a bootstrap approach, we demonstrate a way to quantify the amount of synchrony that goes beyond random coincidence, thus establishing an objective measure for the phenomenon. Applying this technique to psychotherapy data, we develop a hypothesis-driven empirical evaluation of nonverbal synchrony. The method of surrogate testing in order to control for chance is suitable to any corpus of empirical data and lends itself to better empirically informed inference.
---
paper_title: Effects of visual and verbal interaction on unintentional interpersonal coordination

paper_content:

Previous research has demonstrated that people’s movements can become unintentionally coordinated during interpersonal interaction. The current study sought to uncover the degree to which visual and verbal (conversation) interaction constrains and organizes the rhythmic limb movements of coactors. Two experiments were conducted in which pairs of participants completed an interpersonal puzzle task while swinging handheld pendulums with instructions that minimized intentional coordination but facilitated either visual or verbal interaction. Cross-spectral analysis revealed a higher degree of coordination for conditions in which the pairs were visually coupled. In contrast, verbal interaction alone was not found to provide a sufficient medium for unintentional coordination to occur, nor did it enhance the unintentional coordination that emerged during visual interaction. The results raise questions concerning differences between visual and verbal informational linkages during interaction and how these differences may affect interpersonal movement production and its coordination. Interpersonal interaction often results in the movements of two interactants being coordinated. The dyadically defined goals that intentionally constrain interpersonal interaction are typically responsible for the emergence of this coordination. For instance, when two people are dancing, moving a table together, or walking step-by-step with one another, each individual’s behavior is coupled to their partner’s so a shared goal can be achieved in a fluid and unperturbed manner. However, although interpersonal coordination is typically understood as intentional, highly task constrained, and sometimes overtly controlled through physical contact, it is also the case that movement coordination can occur
---
paper_title: Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series

paper_content:

Cross-correlation and most other longitudinal analyses assume that the association between 2 variables is stationary. Thus, a sample of occasions of measurement is expected to be representative of the association between variables regardless of the time of onset or number of occasions in the sample. The authors propose a method to analyze the association between 2 variables when the assumption of stationarity may not be warranted. The method results in estimates of both the strength of peak association and the time lag when the peak association occurred for a range of starting values of elapsed time from the beginning of an experiment.
---
paper_title: Towards visual and vocal mimicry recognition in human-human interactions

paper_content:

During face-to-face interpersonal interaction, people have a tendency to mimic each other. People not only mimic postures, mannerisms, moods or emotions, but they also mimic several speech-related behaviors. In this paper we describe how visual and vocal behavioral information expressed between two interlocutors can be used to detect and identify visual and vocal mimicry. We investigate expressions of mimicry and aim to learn more about in which situation and to what extent mimicry occurs. The observable effects of mimicry can be explored by representing and recognizing mimicry using visual and vocal features. In order to automatically analyze how to extract and integrate this behavioral information into a multimodal mimicry detection framework for improving affective computing, this paper addresses the main challenge: mimicry representation in terms of optimal behavioral feature extraction and automatic integration in both audio and video modalities.
---
paper_title: Automatic visual mimicry expression analysis in interpersonal interaction

paper_content:

Mimicry occurs in conversations both when people agree with each other and when they do not. However, it has been reported that there is more mimicry when people agree than when they disagree: when people want to express shared opinions and attitudes, they do so by displaying behavior that is similar to their interlocutors' behavior. In a conversation, mimicry occurs in order to gain acceptance from an interaction partner by conforming to that person's attitudes, opinions, and behavior. In this paper we describe how visual behavioral information expressed between two interlocutors can be used to detect and identify visual mimicry. We extract and encode visual features that are expected to represent mimicry in a useful way. In order to show that mimicry has indeed occurred, we calculate correlations between visual features extracted from the interactants and compare these against each other and against a baseline. We show that it is possible to visualize the occurrence of visual mimicry during the progress of a conversation which allows us to research in which situations and to what extent mimicry occurs.
---
paper_title: Prosodic and gestural expression of interactional agreement

paper_content:

Conversational interactions are cooperatively constructed activities in which participants negotiate their entrances, turns and alignments with other speakers, oftentimes with an underlying long-term objective of obtaining some agreement. Obtaining a final and morally binding accord in a conversational interaction is of importance in a great variety of contexts, particularly in psychotherapeutic interactions, in contractual negotiations or in educational contexts. Various prosodic and gestural elements in a conversational interaction can be interpreted as signals of a speaker's agreement and they are probably of importance in the emergence of an accord in a conversational exchange. In this paper, we survey the social and psychological context of agreement seeking, as well as the existing literature on the visual and prosodic measurement of agreement in conversational settings.
---
paper_title: Synchrony - a core concept for a constructivist approach to psychotherapy

paper_content:

Synchrony is a pervasive concept relevant to diverse domains in physics, biology, and the social sciences. This article reviews some of the evidence both from natural and human systems. Our main focus is on the role of synchrony in psychotherapy research. Its association with empathy, rapport and the therapeutic relationship has been pointed out repeatedly, yet close evaluation of empirical studies suggests that the evidence remains inconclusive. Particularly in naturalistic studies, research employing synchrony as a specific measure is still lacking. We suggest a new approach to studying synchrony in psychotherapies under field conditions: Image-differencing is a video-based algorithm quantifying the amount of movement in pre-selected regions. Our methodology can be compared to the approach from the seminal work of Condon & Ogston (1966). The statistical analysis we employ detects synchrony on a global level, irrespective of the specific body parts moving. Synchrony thus defined is considered a general measure of coordination of interacting individuals. Exemplary data show the feasibility of this approach and its potential contribution to psychotherapy research.
---
paper_title: Spatiotemporal Symmetry and Multifractal Structure of Head Movements during Dyadic Conversation.

paper_content:

This study examined the influence of sex, social dominance, and context on motion-tracked head movements during dyadic conversations. Windowed cross-correlation analyses found high peak correlation between conversants' head movements over short (≈2-s) intervals and a high degree of nonstationarity. Nonstationarity in head movements was found to be positively related to the number of men in a conversation. Surrogate data analysis offsetting the conversants' time series by a large lag was unable to reject the null hypothesis that the observed high peak correlations were unrelated to short-term coordination between conversants. One way that high peak correlations could be observed when 2 time series are offset by a large time lag is for each time series to exhibit self-similarity over a range of scales. Multifractal analysis found small-scale fluctuations to be persistent, τ(q) 0.5. These results are consistent with a view that symmetry is formed between conversants over short intervals and that this symmetry is broken at longer, irregular intervals.
---
paper_title: Multimodal coordination: exploring relevant features and measures

paper_content:

The decisive role of interpersonal coordination in social interactions is well established. As human beings, we are "experts" in decoding and producing social signals, learning since our birth. Though, equipping a machine with the same abilities represents a great challenge to design socially-accepted interfaces. Assessing automatically this coordination requires selecting the most relevant features. In this paper, various audio and visual features are investigated to characterize interactional synchrony in a cooperative task. A hierarchical representation of the similarity between extracted features is proposed. The properties of time-domain (correlation coefficients) and frequency-domain (coherence) synchrony measures are confronted. The presented results support the evidence that synchrony is a multimodal phenomenon. Moreover, it can be assessed both on local (short-length windows of interaction) or global scales (entire interaction).
---
paper_title: Symmetry building and symmetry breaking in synchronized movement

paper_content:

Mirror symmetry between individuals is a behavioral phenomenon that is commonly observed in conversation: individuals tend to mimic each other’s postures and gestures as a part of a shared dialog. The present work studies the process of symmetry building and symmetry breaking in the movements of pairs of individuals while imitating each others’ movements in dance. Spatial and temporal symmetries are found in the overall velocities from the results of full body motion tracking. This work was supported in part by NIA grant R29 AG14983-01 and a grant from the University of Notre Dame. We wish to thank Elizabeth Marsh who assisted in gathering these data. Corresponding author: Steven M. Boker, Department of Psychology, The University of Notre Dame, Notre Dame Indiana 46556, USA; sboker@nd.edu; http://www.nd.edu/ sboker. SYMMETRY IN SYNCHRONIZED MOVEMENT 2
---
paper_title: Measuring Multimodal Synchrony for Human-Computer Interaction

paper_content:

Nonverbal synchrony is an important and natural element in human-human interaction. It can also play various roles in human-computer interaction. In particular this is the case in the interaction between humans and the virtual humans that inhabit our cyber worlds. Virtual humans need to adapt their behavior to the behavior of their human interaction partners in order to maintain a natural and continuous interaction synchronization. This paper surveys approaches to modeling synchronization and applications where this modeling is important. Apart from presenting this framework, we also present a quantitative method for measuring the level of nonverbal synchrony in an interaction and observations on future research that allows embedding such methods in models of interaction behavior of virtual humans.
---
paper_title: Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension

paper_content:

Looking To Understand: The Coupling Between Speakers’ and Listeners’ Eye Movements and its Relationship to Discourse Comprehension Daniel C. Richardson (richardson@psych.stanford.edu) Department of Psychology, Stanford University Stanford, CA 94305, USA Rick Dale (rad28@cornell.edu) Department of Psychology, Cornell University Ithaca, NY 14853, USA Abstract discussing a diagram drawn on a whiteboard, figuring out together how to do something on a computer, or talking during a movie. Uniquely poised between perception and cognition, eye movements can reveal cognitive processes such as speech planning, language comprehension, memory, mental imagery and decision making. The current experiment investigates whether the eye movements of a speaker and a listener to a visual common ground can provide insight into a discourse. While their eye movements were being recorded, participants spoke extemporaneously about a TV show whose cast members they were viewing. Later, other participants listened to these speeches while their eyes were tracked. Within this naturalistic paradigm using spontaneous speech, a number of results linking eye movements to speech comprehension, speech production and memory were replicated. More importantly, a cross- recurrence analysis demonstrated that speaker and listener eye movements were coupled, and that the strength of this relationship positively correlated with listeners’ comprehension. Just as the mental state of a single person can be reflected in patterns of eye movements, the commonality of mental states that is brought about by successful communication is mirrored in a similarity between speaker and listener’s eye movements. Eye movement Research Introduction Imagine standing in front of a painting, discussing it with a friend. As you talk, your eyes will scan across the image, moving approximately three times a second. They will be drawn by characteristics of the image itself, areas of contrast or detail, as well as features of the objects or people portrayed. Eye movements are driven both by properties of the visual world and processes in a person’s mind. Your gaze might also be influenced by what your friend is saying, what you say in reply, what is thought but not said, and where you agree and disagree. If this is so, what is the relationship between your eye movements and those of your friend? How is that relationship related to the flow of conversation between you? Language use often occurs within rich visual contexts such as this, and the interplay between linguistic processes and visual perception is of increasing interest to psycholinguists and vision researchers (Henderson & Ferreira, 2004). As yet, however, such processes have been limited to experiments that examine the eye movements of the speaker or the listener in isolation. Language use, more often than not, occurs within a richer social context as well. Direct eye contact between conversants plays an interesting, crucial role in coordinating a conversation (Bavelas, Coates, & Johnson, 2002), and in conveying various attitudes or social roles (Argyle & Cook, 1976). The focus of the current experiment, however, is cases such as those introduced at the outset, where conversants are not looking at each other, but at some visual scene that is the topic of the conversation. More common examples might be Eye movements of a speaker If a speaker is asked to describe a simple scene, they will fixate the objects in the order in which they are mentioned, around 900ms before naming them (Griffin & Bock, 2000; Meyer, Sleiderink, & Levelt, 1998). Since such pictures can be identified rapidly, it is argued that during this time speakers are not just retrieving words but selecting and planning which to use. Eye movements of a listener Eye movement research has shown that there is a tight interdependence between speech recognition and visual perception. Eye movements to potential referents for a word can provide evidence for a lexical item being recognized before the word is finished being spoken. The link between visual and linguistic processing can also be seen in eye movements that disambiguate syntactic structures (Tanenhaus, Spivey Knowlton, Eberhard, & Sedivy, 1995) and anticipate the future agents of actions (Kamide, Altmann, & Haywood, 2003). Recent studies of the eye- movements of a participant engaged in a conversation with another naive participant reveal a remarkable sensitivity to the referential domains established by the task, the visual context and the preceding conversation (Brown-Schmidt, Campana, & Tanenhaus, 2004). Qualitatively, eye movement research reveals a very close, time-locked integration between visual and linguistic processing (Tanenhaus, Magnuson, Dahan, & Chambers, 2000). Although fixation times are heavily modulated by context, as a very rough quantitative guide, research suggests that listeners will fixate an object around 400-800ms after the name onset.
---
paper_title: Social coordination dynamics: Measuring human bonding

paper_content:

Abstract Spontaneous social coordination has been extensively described in natural settings but so far no controlled methodological approaches have been employed that systematically advance investigations into the possible self-organized nature of bond formation and dissolution between humans. We hypothesized that, under certain contexts, spontaneous synchrony—a well-described phenomenon in biological and physical settings—could emerge spontaneously between humans as a result of information exchange. Here, a new way to quantify interpersonal interactions in real time is proposed. In a simple experimental paradigm, pairs of participants facing each other were required to actively produce actions, while provided (or not) with the vision of similar actions being performed by someone else. New indices of interpersonal coordination, inspired by the theoretical framework of coordination dynamics (based on relative phase and frequency overlap between movements of individuals forming a pair) were developed and us...
---
paper_title: Multimodal coordination: exploring relevant features and measures

paper_content:

The decisive role of interpersonal coordination in social interactions is well established. As human beings, we are "experts" in decoding and producing social signals, learning since our birth. Though, equipping a machine with the same abilities represents a great challenge to design socially-accepted interfaces. Assessing automatically this coordination requires selecting the most relevant features. In this paper, various audio and visual features are investigated to characterize interactional synchrony in a cooperative task. A hierarchical representation of the similarity between extracted features is proposed. The properties of time-domain (correlation coefficients) and frequency-domain (coherence) synchrony measures are confronted. The presented results support the evidence that synchrony is a multimodal phenomenon. Moreover, it can be assessed both on local (short-length windows of interaction) or global scales (entire interaction).
---
paper_title: Effects of visual and verbal interaction on unintentional interpersonal coordination

paper_content:

Previous research has demonstrated that people’s movements can become unintentionally coordinated during interpersonal interaction. The current study sought to uncover the degree to which visual and verbal (conversation) interaction constrains and organizes the rhythmic limb movements of coactors. Two experiments were conducted in which pairs of participants completed an interpersonal puzzle task while swinging handheld pendulums with instructions that minimized intentional coordination but facilitated either visual or verbal interaction. Cross-spectral analysis revealed a higher degree of coordination for conditions in which the pairs were visually coupled. In contrast, verbal interaction alone was not found to provide a sufficient medium for unintentional coordination to occur, nor did it enhance the unintentional coordination that emerged during visual interaction. The results raise questions concerning differences between visual and verbal informational linkages during interaction and how these differences may affect interpersonal movement production and its coordination. Interpersonal interaction often results in the movements of two interactants being coordinated. The dyadically defined goals that intentionally constrain interpersonal interaction are typically responsible for the emergence of this coordination. For instance, when two people are dancing, moving a table together, or walking step-by-step with one another, each individual’s behavior is coupled to their partner’s so a shared goal can be achieved in a fluid and unperturbed manner. However, although interpersonal coordination is typically understood as intentional, highly task constrained, and sometimes overtly controlled through physical contact, it is also the case that movement coordination can occur
---
paper_title: Social Coordination, from the Perspective of Coordination Dynamics.

paper_content:

PCT No. PCT/KR97/00282 Sec. 371 Date Jan. 28, 1999 Sec. 102(e) Date Jan. 28, 1999 PCT Filed Dec. 29, 1997 PCT Pub. No. WO98/44532 PCT Pub. Date Oct. 8, 1998An alternating current plasma display panel with a surface discharge. The panel comprises in each pixel three parallel electrodes, one of the three parallel electrodes being an address electrode, and drive electrodes being perpendicular to them, the three parallel electrodes, the three parallel electrodes, and the drive electrode being separated by dielectric barrier ribs, geometric parameters of electrodes, discharge gaps and gas filling being determined from conditions needed for execution of a surface discharge by sustaining voltage pulses in a pixel between two extreme parallel electrodes with the address electrode arranged between them. When the discharge gap length is increased in the luminescence area, power does not increase but brightness is improved. The ratio of brightness of new/standard and the ratio of power new/standard becomes 1.1-2.3 times better.
---
paper_title: Automatic visual mimicry expression analysis in interpersonal interaction

paper_content:

Mimicry occurs in conversations both when people agree with each other and when they do not. However, it has been reported that there is more mimicry when people agree than when they disagree: when people want to express shared opinions and attitudes, they do so by displaying behavior that is similar to their interlocutors' behavior. In a conversation, mimicry occurs in order to gain acceptance from an interaction partner by conforming to that person's attitudes, opinions, and behavior. In this paper we describe how visual behavioral information expressed between two interlocutors can be used to detect and identify visual mimicry. We extract and encode visual features that are expected to represent mimicry in a useful way. In order to show that mimicry has indeed occurred, we calculate correlations between visual features extracted from the interactants and compare these against each other and against a baseline. We show that it is possible to visualize the occurrence of visual mimicry during the progress of a conversation which allows us to research in which situations and to what extent mimicry occurs.
---
paper_title: Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension

paper_content:

Looking To Understand: The Coupling Between Speakers’ and Listeners’ Eye Movements and its Relationship to Discourse Comprehension Daniel C. Richardson (richardson@psych.stanford.edu) Department of Psychology, Stanford University Stanford, CA 94305, USA Rick Dale (rad28@cornell.edu) Department of Psychology, Cornell University Ithaca, NY 14853, USA Abstract discussing a diagram drawn on a whiteboard, figuring out together how to do something on a computer, or talking during a movie. Uniquely poised between perception and cognition, eye movements can reveal cognitive processes such as speech planning, language comprehension, memory, mental imagery and decision making. The current experiment investigates whether the eye movements of a speaker and a listener to a visual common ground can provide insight into a discourse. While their eye movements were being recorded, participants spoke extemporaneously about a TV show whose cast members they were viewing. Later, other participants listened to these speeches while their eyes were tracked. Within this naturalistic paradigm using spontaneous speech, a number of results linking eye movements to speech comprehension, speech production and memory were replicated. More importantly, a cross- recurrence analysis demonstrated that speaker and listener eye movements were coupled, and that the strength of this relationship positively correlated with listeners’ comprehension. Just as the mental state of a single person can be reflected in patterns of eye movements, the commonality of mental states that is brought about by successful communication is mirrored in a similarity between speaker and listener’s eye movements. Eye movement Research Introduction Imagine standing in front of a painting, discussing it with a friend. As you talk, your eyes will scan across the image, moving approximately three times a second. They will be drawn by characteristics of the image itself, areas of contrast or detail, as well as features of the objects or people portrayed. Eye movements are driven both by properties of the visual world and processes in a person’s mind. Your gaze might also be influenced by what your friend is saying, what you say in reply, what is thought but not said, and where you agree and disagree. If this is so, what is the relationship between your eye movements and those of your friend? How is that relationship related to the flow of conversation between you? Language use often occurs within rich visual contexts such as this, and the interplay between linguistic processes and visual perception is of increasing interest to psycholinguists and vision researchers (Henderson & Ferreira, 2004). As yet, however, such processes have been limited to experiments that examine the eye movements of the speaker or the listener in isolation. Language use, more often than not, occurs within a richer social context as well. Direct eye contact between conversants plays an interesting, crucial role in coordinating a conversation (Bavelas, Coates, & Johnson, 2002), and in conveying various attitudes or social roles (Argyle & Cook, 1976). The focus of the current experiment, however, is cases such as those introduced at the outset, where conversants are not looking at each other, but at some visual scene that is the topic of the conversation. More common examples might be Eye movements of a speaker If a speaker is asked to describe a simple scene, they will fixate the objects in the order in which they are mentioned, around 900ms before naming them (Griffin & Bock, 2000; Meyer, Sleiderink, & Levelt, 1998). Since such pictures can be identified rapidly, it is argued that during this time speakers are not just retrieving words but selecting and planning which to use. Eye movements of a listener Eye movement research has shown that there is a tight interdependence between speech recognition and visual perception. Eye movements to potential referents for a word can provide evidence for a lexical item being recognized before the word is finished being spoken. The link between visual and linguistic processing can also be seen in eye movements that disambiguate syntactic structures (Tanenhaus, Spivey Knowlton, Eberhard, & Sedivy, 1995) and anticipate the future agents of actions (Kamide, Altmann, & Haywood, 2003). Recent studies of the eye- movements of a participant engaged in a conversation with another naive participant reveal a remarkable sensitivity to the referential domains established by the task, the visual context and the preceding conversation (Brown-Schmidt, Campana, & Tanenhaus, 2004). Qualitatively, eye movement research reveals a very close, time-locked integration between visual and linguistic processing (Tanenhaus, Magnuson, Dahan, & Chambers, 2000). Although fixation times are heavily modulated by context, as a very rough quantitative guide, research suggests that listeners will fixate an object around 400-800ms after the name onset.
---
paper_title: Spatiotemporal Symmetry and Multifractal Structure of Head Movements during Dyadic Conversation.

paper_content:

This study examined the influence of sex, social dominance, and context on motion-tracked head movements during dyadic conversations. Windowed cross-correlation analyses found high peak correlation between conversants' head movements over short (≈2-s) intervals and a high degree of nonstationarity. Nonstationarity in head movements was found to be positively related to the number of men in a conversation. Surrogate data analysis offsetting the conversants' time series by a large lag was unable to reject the null hypothesis that the observed high peak correlations were unrelated to short-term coordination between conversants. One way that high peak correlations could be observed when 2 time series are offset by a large time lag is for each time series to exhibit self-similarity over a range of scales. Multifractal analysis found small-scale fluctuations to be persistent, τ(q) 0.5. These results are consistent with a view that symmetry is formed between conversants over short intervals and that this symmetry is broken at longer, irregular intervals.
---
paper_title: Multimodal coordination: exploring relevant features and measures

paper_content:

The decisive role of interpersonal coordination in social interactions is well established. As human beings, we are "experts" in decoding and producing social signals, learning since our birth. Though, equipping a machine with the same abilities represents a great challenge to design socially-accepted interfaces. Assessing automatically this coordination requires selecting the most relevant features. In this paper, various audio and visual features are investigated to characterize interactional synchrony in a cooperative task. A hierarchical representation of the similarity between extracted features is proposed. The properties of time-domain (correlation coefficients) and frequency-domain (coherence) synchrony measures are confronted. The presented results support the evidence that synchrony is a multimodal phenomenon. Moreover, it can be assessed both on local (short-length windows of interaction) or global scales (entire interaction).
---
paper_title: Nonverbal synchrony or random coincidence? how to tell the difference

paper_content:

Nonverbal synchrony in face-to-face interaction has been studied in numerous empirical investigations focusing on various communication channels. Furthermore, the pervasiveness of synchrony in physics, chemistry and biology adds to its face-validity. This paper is focused on establishing criteria for a statistical evaluation of synchrony in human interaction. When assessing synchrony in any communication context, it is necessary to distinguish genuine synchrony from pseudosynchrony, which may arise due to random coincidence. By using a bootstrap approach, we demonstrate a way to quantify the amount of synchrony that goes beyond random coincidence, thus establishing an objective measure for the phenomenon. Applying this technique to psychotherapy data, we develop a hypothesis-driven empirical evaluation of nonverbal synchrony. The method of surrogate testing in order to control for chance is suitable to any corpus of empirical data and lends itself to better empirically informed inference.
---
paper_title: Discovering hidden time patterns in behavior: T-patterns and their detection

paper_content:

This article deals with the definition and detection of particular kinds of temporal patterns in behavior, which are sometimes obvious or well known, but other times difficult to detect, either directly or with standard statistical methods. Characteristics of well-known behavior patterns were abstracted and combined in order to define a scale-independent, hierarchical time pattern type, called aT-pattern. A corresponding detection algorithm was developed and implemented in a computer program, called Theme. The proposed pattern typology and detection algorithm are based on the definition and detection of a particular relationship between pairs of events in a time series, called acritical interval relation. The proposed bottom-up, level-by-level (or breadth-first) search algorithm is based on a binary tree of such relations. The algorithm first detects simpler patterns. Then, more complex and complete patterns evolve through the connection of simpler ones, pattern completeness competition, and pattern selection. Interindividual T-patterns in a quarter-hour interaction between two children are presented, showing that complex hidden T-patterns may be found by Theme in such behavioral streams. Finally, implications for studies of complexity, self-organization, and dynamic patterns are discussed.
---
paper_title: Cluster Ensembles – A Knowledge Reuse Framework for Combining Multiple Partitions

paper_content:

This paper introduces the problem of combining multiple partitionings of a set of objects into a single consolidated clustering without accessing the features or algorithms that determined these partitionings. We first identify several application scenarios for the resultant 'knowledge reuse' framework that we call cluster ensembles. The cluster ensemble problem is then formalized as a combinatorial optimization problem in terms of shared mutual information. In addition to a direct maximization approach, we propose three effective and efficient techniques for obtaining high-quality combiners (consensus functions). The first combiner induces a similarity measure from the partitionings and then reclusters the objects. The second combiner is based on hypergraph partitioning. The third one collapses groups of clusters into meta-clusters which then compete for each object to determine the combined clustering. Due to the low computational costs of our techniques, it is quite feasible to use a supra-consensus function that evaluates all three approaches against the objective function and picks the best solution for a given situation. We evaluate the effectiveness of cluster ensembles in three qualitatively different application scenarios: (i) where the original clusters were formed based on non-identical sets of features, (ii) where the original clustering algorithms worked on non-identical sets of objects, and (iii) where a common data-set is used and the main purpose of combining multiple clusterings is to improve the quality and robustness of the solution. Promising results are obtained in all three situations for synthetic as well as real data-sets.
---
paper_title: Understanding Parent-Infant Behaviors Using Non-negative Matrix Factorization

paper_content:

There are considerable differences among infants in the quality of interaction with their parents. These differences depend especially on the infants development which affects the parent behaviors. In our study we are interested on 3 groups of infants: typical development infants (TD), autistic infants (AD) and mental retardation infants (MR). In order to identify the groups of signs/behaviors that differentiate the development of the three groups of children we investigated a clustering method NMF (non-negative matrix factorization), an algorithm based on decomposition by parts that can reduce the dimension of interaction signs to a few number of interaction behaviors groups. Coupled with a statistical data representation tf-idf, usually used for document clustering and adapted to our work, NMF provides an efficient method for identification of distinct interaction groups. Forty-two infants and their parents were observed in this study. Parent-infant interactions were videotaped by one of the parents at home during the first two years age of child.
---
paper_title: THE COURTSHIP DANCE: PATTERNS OF NONVERBAL SYNCHRONIZATION IN OPPOSITE-SEX ENCOUNTERS

paper_content:

This study examines the existence of behavioral correlates of synchronization on different levels of analysis and methods. We were unable to demonstrate a relation between synchronization defined in terms of movement echo or position mirroring and subjective experience of pleasure and interest in opposite-sex encounters. Significant results were found for a phenomenon we describe as hierarchically patterned synchronization. These patterns were identified with the help of a newly developed search algorithm. If a female is interested in a male, highly complex patterns of behavior with a constant time structure emerge. The patterns are pair-specific and independent from behavioral content. This rhythmic structure of interactions is discussed in functional terms of human courtship.
---
paper_title: Positive matrix factorization: A non‐negative factor model with optimal utilization of error estimates of data values

paper_content:

A new variant ‘PMF’ of factor analysis is described. It is assumed that X is a matrix of observed data and σ is the known matrix of standard deviations of elements of X. Both X and σ are of dimensions n × m. The method solves the bilinear matrix problem X = GF + E where G is the unknown left hand factor matrix (scores) of dimensions n × p, F is the unknown right hand factor matrix (loadings) of dimensions p × m, and E is the matrix of residuals. The problem is solved in the weighted least squares sense: G and F are determined so that the Frobenius norm of E divided (element-by-element) by σ is minimized. Furthermore, the solution is constrained so that all the elements of G and F are required to be non-negative. It is shown that the solutions by PMF are usually different from any solutions produced by the customary factor analysis (FA, i.e. principal component analysis (PCA) followed by rotations). Usually PMF produces a better fit to the data than FA. Also, the result of PF is guaranteed to be non-negative, while the result of FA often cannot be rotated so that all negative entries would be eliminated. Different possible application areas of the new method are briefly discussed. In environmental data, the error estimates of data can be widely varying and non-negativity is often an essential feature of the underlying models. Thus it is concluded that PMF is better suited than FA or PCA in many environmental applications. Examples of successful applications of PMF are shown in companion papers.
---
paper_title: A Multimodal End-of-Turn Prediction Model: Learning from Parasocial Consensus Sampling

paper_content:

human, with realistic behaviors and social skills, evoke in users a range of social behaviors normally only seen in human face-to-face interactions. One of the key challenges in creating such virtual humans is to give them human-like conversational skills, such as turn-taking skill. In this paper, we propose a multimodal end-of-turn prediction model. Instead of recording face-to-face conversation data, we collect the turn-taking data using Parasocial Consensus Sampling (PCS) framework. Then we analyze the relationship between verbal and nonverbal features and turn-taking behaviors based on the consensus data and show how these features influence the time people use to take turns. Finally, we present a probabilistic multimodal end-of- turn prediction model, which enables virtual humans to make real-time turn-taking predictions. The result shows that our model achieves a higher accuracy than previous methods did.
---
paper_title: Latent Mixture of Discriminative Experts for Multimodal Prediction Modeling

paper_content:

During face-to-face conversation, people naturally integrate speech, gestures and higher level language interpretations to predict the right time to start talking or to give backchannel feedback. In this paper we introduce a new model called Latent Mixture of Discriminative Experts which addresses some of the key issues with multimodal language processing: (1) temporal synchrony/asynchrony between modalities, (2) micro dynamics and (3) integration of different levels of interpretation. We present an empirical evaluation on listener nonverbal feedback prediction (e.g., head nod), based on observable behaviors of the speaker. We confirm the importance of combining four types of multimodal features: lexical, syntactic structure, eye gaze, and prosody. We show that our Latent Mixture of Discriminative Experts model outperforms previous approaches based on Conditional Random Fields (CRFs) and Latent-Dynamic CRFs.
---
paper_title: Dialog Prediction for a General Model of Turn-Taking

paper_content:

Today there are solutions for some specific turn-taking problems, but no general model. We show how turn-taking can be reduced to two more general problems, prediction and selection. We also discuss the value of predicting not only future speech/silence but also prosodic features, thereby handing not only turn-taking but “turn-shaping”. To illustrate how such predictions can be made, we trained a neural network predictor. This was adequate to support some specific turn-taking decisions and was modestly accurate overall.
---
paper_title: Natural Turn-Taking Needs No Manual: Computational Theory and Model, from Perception to Action

paper_content:

Decisions like these are made by dialogue participants as often as 2–3 times per second. For a 30 minute conversation that’s over 5000 decisions. And that’s just a fraction of what goes on. How do we do it? Face-to-face dialogue consists of interaction between several complex, dynamic systems — visual and auditory display of information, internal processing, knee-jerk reactions, thought-out rhetoric, learned patterns, social convention, etc. One could postulate that the power of dialogue is a direct result of this fact. However, combining a multitude of systems in one place does not guarantee a coherent outcome such as goal-directed dialogue. For this to happen the systems need to be architected in a way that guides their interaction and ensures that — complex as it may be — the interaction tends towards homeostasis in light of errors and uncertainties, towards the set of goals shared by participants.
---
paper_title: Predicting Speaker Changes and Listener Responses With And Without Eye-contact

paper_content:

This paper compares turn-taking in terms of timing and prediction in human-human conversations under the conditions when participants has eye-contact versus when there is no eyecontact, as found in ...
---
paper_title: Backchannel-Inviting Cues in Task-Oriented Dialogue

paper_content:

We examine BACKCHANNEL-INVITING CUES — distinct prosodic, acoustic and lexical events in the speaker’s speech that tend to precede a short response produced by the interlocutor to convey continued attention —in theColumbiaGames Corpus, a large corpus of task-oriented dialogues. We show that the likelihood of occurrence of a backchannel increases quadratically with the number of cues conjointly displayed by the speaker. Our results are important for improving the coordination of conversational turns in interactive voice-response systems, so that systems can produce backchannels in appropriate places, and so that they can elicit backchannels from users in expected places. Index Terms: dialogue, prosody, turn-taking, backchannels.
---
paper_title: Predicting interruptions in dyadic spoken interactions

paper_content:

Interruptions occur frequently in spontaneous conversations, and they are often associated with changes in the flow of conversation. Predicting interruption is essential in the design of natural human-machine spoken dialog interface. The modeling can bring insights into the dynamics of human-human conversation. This work utilizes Hidden Condition Random Field (HCRF) to predict occurrences of interruption in dyadic spoken interactions by modeling both speakers' behaviors before a turn change takes place. Our prediction model, using both the foreground speaker's acoustic cues and the listener's gestural cues, achieves an F-measure of 0.54, accuracy of 70.68%, and unweighted accuracy of 66.05% on a multimodal database of dyadic interactions. The experimental results also show that listener's behaviors provides an indication of his/her intention of interruption.
---
paper_title: A shallow model of backchannel continuers in spoken dialogue

paper_content:

Spoken dialogue systems would be more acceptable if they were able to produce backchannel continuers such as mm-hmm in naturalistic locations during the user's utterances. Using the HCRC Map Task Corpus as our data source, we describe models for predicting these locations using only limited processing and features of the user's speech that are commonly available, and which therefore could be used as a low-cost improvement for current systems. The baseline model inserts continuers after a predetermined number of words. One further model correlates back-channel continuers with pause duration, while a second predicts their occurrence using trigram POS frequencies. Combining these two models gives the best results.
---
paper_title: Predicting listener backchannels: A probabilistic multimodal approach

paper_content:

During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model or Conditional Random Fields) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.
---
paper_title: Movement coordination in social interaction: Some examples described

paper_content:

Abstract Sound films of social interaction are analysed and some detailed descriptions of the interrelations of the movements of speakers ans listeners are given. These descriptions provide further examples of ‘interactional synchrony’, first described by Condon and Ogston (1966). It is found that the flow of movement in the listener may be rhythmically coordinated with the speech and movements of the speaker. It is also shown how the way in which individuals may be in synchrony with one another can vary, and that these variations are related to their respective roles in the interaction. Some of the implications of the phenomenon of interactional synchrony are discussed.
---
paper_title: A multimodal database for mimicry analysis

paper_content:

In this paper we introduce a multi-modal database for the analysis of human interaction, in particular mimicry, and elaborate on the theoretical hypotheses of the relationship between the occurrence of mimicry and human affect. The recorded experiments are designed to explore this relationship. The corpus is recorded with 18 synchronised audio and video sensors, and is annotated for many different phenomena, including dialogue acts, turn-taking, affect, head gestures, hand gestures, body movement and facial expression. Recordings were made of two experiments: a discussion on a political topic, and a role-playing game. 40 participants were recruited, all of whom selfreported their felt experiences. The corpus will be made available to the scientific community.
---
paper_title: Dyadic synchrony: Its structure and function in children’s development ☆

paper_content:

In this review we examine empirical and theoretical work in three eras—infancy, toddlerhood, and early childhood—and for each era describe the structure of dyadic synchrony in interactions involving children and their caregivers, as well as offer speculation about its developmental function for the child. We review divergent literatures dealing with synchrony-related constructs which, together, suggest that although the structure and function of synchrony change throughout the course of early development, the ability to achieve synchrony may represent a crucial developmental achievement for significant dyadic relationships, one that facilitates social, emotional, and cognitive growth for the child.
---
paper_title: Movement synchrony and perceived entitativity

paper_content:

Movement synchrony has been theoretically linked to the emergence of a social unit. To empirically investigate whether similar movement rhythms are an antecedent of perceived entitativity, movement rhythms were experimentally manipulated in four studies. Using this novel approach, stick figures waving in synchrony were found to be rated higher on entitativity than stick figures waving in different rhythms (Study 1), and this effect was extended to interactional synchrony, where different movements are performed in the same rhythm (Study 2). Objective differences in movement rhythms are linearly related to ratings of perceived entitativity, and this relationship is partially mediated by the subjectively perceived similarity of movement rhythms (Study 3). These results also held for entitativity judgments for videotaped individuals waving rhythmically (Study 4). These results support the hypothesis that movement rhythms are an important source of information which observers use to infer the extent to which individuals are a social unit.
---
paper_title: An Audio-Visual Approach to Measuring Discourse Synchrony in Multimodal Conversation Data

paper_content:

This paper describes recent work on the automatic extraction of visual and audio parameters relating to the detection of synchrony in discourse, and to the modelling of active listening for advanced speech technology. It reports findings based on image processing that reliably identify the strong entrainment between members of a group conversation, and describes techniques for the extraction and analysis of such information.
---
paper_title: Experiencing contingency and agency: First step toward self-understanding in making a mind?

paper_content:

Precursors of inferential capacities concerning self- and other- understanding may be found in the basic experience of social contingency and emotional sharing. The emergence of a sense of self- and other-agency receives special attention here, as a foundation for self-understanding. We propose that synchrony, an amodal parameter of contingent self-other relationships, should be especially involved in the development of a sense of agency. To explore this framework, we have manipulated synchrony in various ways, either by delaying mother’s response to infant’s behaviour, disorganizing mother’s internal synchrony between face and voice, freezing the partner in a still attitude, or on the contrary maximizing synchrony through imitation. We report results obtained with healthy and clinical populations that are supposed to be at the beginning of basic experiences concerning the ownership of their actions: infants of 2 months and 6 months, low-functioning children with autism and MA matched young children with Down Syndrome. Our results support the idea of a two-step process linking understanding of self to understanding of other and leading on to form the concept of human beings as universally contingent entities.
---
paper_title: Characterization of coordination in an imitation task: human evaluation and automatically computable cues

paper_content:

Understanding the ability to coordinate with a partner constitutes a great challenge in social signal processing and social robotics. In this paper, we designed a child-adult imitation task to investigate how automatically computable cues on turn-taking and movements can give insight into high-level perception of coordination. First we collected a human questionnaire to evaluate the perceived coordination of the dyads. Then, we extracted automatically computable cues and information on dialog acts from the video clips. The automatic cues characterized speech and gestural turn-takings and coordinated movements of the dyad. We finally confronted human scores with automatic cues to search which cues could be informative on the perception of coordination during the task. We found that the adult adjusted his behavior according to the child need and that a disruption of the gestural turn-taking rhythm was badly perceived by the judges. We also found, that judges rated negatively the dyads that talked more as speech intervenes when the child had difficulties to imitate. Finally, coherence measures between the partners' movement features seemed more adequate than correlation to characterize their coordination.
---
paper_title: Integration of Action and Language Knowledge: A Roadmap for Developmental Robotics

paper_content:

This position paper proposes that the study of embodied cognitive agents, such as humanoid robots, can advance our understanding of the cognitive development of complex sensorimotor, linguistic, and social learning skills. This in turn will benefit the design of cognitive robots capable of learning to handle and manipulate objects and tools autonomously, to cooperate and communicate with other robots and humans, and to adapt their abilities to changing internal, environmental, and social conditions. Four key areas of research challenges are discussed, specifically for the issues related to the understanding of: 1) how agents learn and represent compositional actions; 2) how agents learn and represent compositional lexica; 3) the dynamics of social interaction and learning; and 4) how compositional action and language representations are integrated to bootstrap the cognitive system. The review of specific issues and progress in these areas is then translated into a practical roadmap based on a series of milestones. These milestones provide a possible set of cognitive robotics goals and test scenarios, thus acting as a research roadmap for future work on cognitive developmental robotics.
---
paper_title: Foundations for a new science of learning

paper_content:

Human learning is distinguished by the range and complexity of skills that can be learned and the degree of abstraction that can be achieved compared with those of other species. Homo sapiens is also the only species that has developed formal ways to enhance learning: teachers, schools, and curricula. Human infants have an intense interest in people and their behavior and possess powerful implicit learning mechanisms that are affected by social interaction. Neuroscientists are beginning to understand the brain mechanisms underlying learning and how shared brain systems for perception and action support social learning. Machine learning algorithms are being developed that allow robots and computers to learn autonomously. New insights from many different fields are converging to create a new science of learning that may transform educational practices.
---
paper_title: How an agent can detect and use synchrony parameter of its own interaction with a human

paper_content:

Synchrony is claimed by psychology as a crucial parameter of any social interaction: to give to human a feeling of natural interaction, a feeling of agency [17], an agent must be able to synchronise with this human on appropriate time [29] [11] [15] [16] [27]. In the following experiment, we show that synchrony can be more than a state to reach during interaction, it can be a useable cue of the human’s satisfaction and level of engagement concerning the ongoing interaction: the better is the interaction, the more synchronous with the agent is the human. We built an architecture that can acquire a human partner’s level of synchrony and use this parameter to adapt the agent behavior. This architecture detects temporal relation [1] existing between the actions of the agent and the actions of the human. We used this detected level of synchrony as reinforcement for learning [6]: the more constant the temporal relation between agent and human remains, the more positive is the reinforcement, conversely if the temporal relation varies above a threshold the reinforcement is negative. In a teaching task, this architecture enables naive humans to make the agent learn left-right associations just by the mean of intuitive interactions. The convergence of this learning reinforced by synchrony shows that synchrony conveys current information concerning human satisfaction and that we are able to extract and reuse this information to adapt the agent behavior appropriately.
---
paper_title: Rhythmic attention in child-robot dance play

paper_content:

Human social behavior is rhythmic, and synchrony plays an important role in coordinating and regulating our interactions. We are developing technology that allows the robot Keepon to perceive and behave rhythmically, and to synchronize its dancing behaviors to music or to children's movement as perceived using pressure sensors. We present two experiments in which Keepon dances with children to music, and in which the robot's rhythmic attention and role of leader or follower are manipulated in order to examine the effects on engagement and rhythmic synchrony. We found that children can assume the roles of leader or follower in a rhythmic interaction, that followers indeed tend to synchronize with the robot's movements, and that the role of follower causes the children to more closely follow a musical rhythm.
---
paper_title: Creating Rapport with Virtual Agents

paper_content:

Recent research has established the potential for virtual characters to establish rapport with humans through simple contingent nonverbal behaviors. We hypothesized that the contingency, not just the frequency of positive feedback is crucial when it comes to creating rapport. The primary goal in this study was evaluative: can an agent generate behavior that engenders feelings of rapport in human speakers and how does this compare to human generated feedback? A secondary goal was to answer the question: Is contingency (as opposed to frequency) of agent feedback crucial when it comes to creating feelings of rapport? Results suggest that contingency matters when it comes to creating rapport and that agent generated behavior was as good as human listeners in creating rapport. A "virtual human listener" condition performed worse than other conditions.
---
paper_title: Early language acquisition: cracking the speech code

paper_content:

Infants learn language with remarkable speed, but how they do it remains a mystery. New data show that infants use computational strategies to detect the statistical and prosodic patterns in language input, and that this leads to the discovery of phonemes and words. Social interaction with another human being affects speech learning in a way that resembles communicative learning in songbirds. The brain's commitment to the statistical and prosodic patterns that are experienced early in life might help to explain the long-standing puzzle of why infants are better language learners than adults. Successful learning by infants, as well as constraints on that learning, are changing theories of language acquisition.
---
paper_title: ‘ ‘ Social ’ ’ robots are psychological agents for infants : A test of gaze following

paper_content:

Gaze following is a key component of human social cognition. Gaze following directs attention to areas of high information value and accelerates social, causal, and cultural learning. An issue for both robotic and infant learning is whose gaze to follow. The hypothesis tested in this study is that infants use information derived from an entity's interactions with other agents as evidence about whether that entity is a perceiver. A robot was programmed so that it could engage in communicative, imitative exchanges with an adult experimenter. Infants who saw the robot act in this social-communicative fashion were more likely to follow its line of regard than those without such experience. Infants use prior experience with the robot's interactions as evidence that the robot is a psychological agent that can see. Infants want to look at what the robot is seeing, and thus shift their visual attention to the external target.
---
paper_title: Probabilistic epigenesis: An alternative causal model for conduct disorders in children and adolescents

paper_content:

Most models of conduct disorders are based on a biopsychosocial perspective. In such models, genetic and/or neuropsychological determinants are assumed to have a core hierarchical role by conferring vulnerabilities during a child's development. Based on a review of recent literature, I will show that other views are attempting to integrate the diversity and complexity of factors that influence these behavioural manifestations. The model that best integrates the complexity of these different factors is a developmental model of probabilistic epigenesis. In this paper, I summarise several lines of study that support this model. In addition to the categorical approach of psychiatric nosography, I will adopt a dimensional approach of phenomenology that will help to follow and focus on specific pertinent dimensions such as clinical signs, psychological characteristics, environmental and/or genetic factors in order to study their interactions and their possible developmental outcomes. This model also permits a specific temporal focus on early interactions that many authors consider crucial in terms of developmental cues.
---
paper_title: Playing with virtual peers: bootstrapping contingent discourse in children with autism

paper_content:

In this paper, we describe an intervention for children with social and communication deficits, such as autism, based on the use of a virtual peer that can engage in tightly collaborative narrative. We present a study in which children with autism engage in collaborative narrative with both a virtual and a human peer, and the use of contingent discourse is compared. Our findings suggest that contingent discourse increased over the course of interaction with a virtual peer, but not a human peer. Furthermore, topic management, such as introducing new topics or maintaining the current topic, was more likely to occur with the virtual peer than with the human peer. We discuss general implications of our work for understanding the role of peer interactions in learning.
---
paper_title: The Clinical Use of Robots for Individuals with Autism Spectrum Disorders: A Critical Review.

paper_content:

Abstract We examined peer-reviewed studies in order to understand the current status of empirically based evidence on the clinical applications of robots in the diagnosis and treatment of Autism Spectrum Disorders (ASD). Studies are organized into four broad categories: (a) the response of individuals with ASD to robots or robot-like behavior in comparison to human behavior, (b) the use of robots to elicit behaviors, (c) the use of robots to model, teach, and/or practice a skill, and (d) the use of robots to provide feedback on performance. A critical review of the literature revealed that most of the findings are exploratory and have methodological limitations that make it difficult to draw firm conclusions about the clinical utility of robots. Finally, we outline the research needed to determine the incremental validity of this technique.
---
paper_title: Probabilistic automata as a model for epigenesis of cellular networks.

paper_content:

Probabilistic automata are compared with deterministic ones in simulations of growing networks made of dividing interconnected cells. On examples of chains, wheels and tree-like structures made of large numbers of cells it is shown that the number of necessary states in the initial generating cell automaton is reduced drastically when the automaton is probabilistic rather than deterministic. Since the price being paid is a decrease in the accuracy of the generated network, conditions under which reasonable compromises can be achieved are studied. They depend on the degree of redundancy of the final network (defined from the complexity of a deterministic automaton capable of generating it with maximum accuracy), on the "entropy" of the generating probabilistic automaton, and on the effects of different inputs on its transition probabilities (as measured by its "'capacity" in the sense of Shannon's information theory). The results are used to discuss and make more precise the notion of biological specificity. It is suggested that the weak metaphor of a genetic program, classically used to account for the role of DNA in specific genetic determinations, is replaced by that of inputs to biochemical probabilistic automata.
---
paper_title: Nonverbal synchrony in psychotherapy: Coordinated body movement reflects relationship quality and outcome.

paper_content:

Objective: The authors quantified nonverbal synchrony—the coordination of patient's and therapist's movement—in a random sample of same-sex psychotherapy dyads. The authors contrasted nonverbal synchrony in these dyads with a control condition and assessed its association with session-level and overall psychotherapy outcome. Method: Using an automated objective video analysis algorithm (Motion Energy Analysis; MEA), the authors calculated nonverbal synchrony in (n = 104) videotaped psychotherapy sessions from 70 Caucasian patients (37 women, 33 men, mean age = 36.5 years, SD = 10.2) treated at an outpatient psychotherapy clinic. The sample was randomly drawn from an archive (N = 301) of routinely videotaped psychotherapies. Patients and their therapists assessed session impact with self-report postsession questionnaires. A battery of pre- and postsymptomatology questionnaires measured therapy effectiveness. Results: The authors found that nonverbal synchrony is higher in genuine interactions contrasted with pseudointeractions (a control condition generated by a specifically designed shuffling procedure). Furthermore, nonverbal synchrony is associated with session-level process as well as therapy outcome: It is increased in sessions rated by patients as manifesting high relationship quality and in patients experiencing high self-efficacy. Higher nonverbal synchrony characterized psychotherapies with higher symptom reduction. Conclusions: The results suggest that nonverbal synchrony embodies the patients' self-reported quality of the relationship and further variables of therapy process. This hitherto overlooked facet of therapeutic relationships might prove useful as an indicator of therapy progress and outcome.
---
paper_title: An Analysis of PCA-based Vocal Entrainment Measures in Married Couples' Affective Spoken Interactions

paper_content:

Entrainment has played a crucial role in analyzing marital couples interactions. In this work, we introduce a novel technique for quantifying vocal entrainment based on Principal Component Analysis (PCA). The entrainment measure, as we define in this work, is the amount of preserved variability of one interlocutor’s speaking characteristic when projected onto representing space of the other’s speaking characteristics. Our analysis on real couples interactions shows that when a spouse is rated as having positive emotion, he/she has a higher value of vocal entrainment compared when rated as having negative emotion. We further performed various statistical analyses on the strength and the directionality of vocal entrainment under different affective interaction conditions to bring quantitative insights into the entrainment phenomenon. These analyses along with a baseline prediction model demonstrate the validity and utility of the proposed PCA-based vocal entrainment measure. Index Terms: vocal entrainment, couples therapy, behavioral signal processing, principal component analysis
---
paper_title: Advances in Behavioral Science Using Automated Facial Image Analysis and Synthesis [Social Sciences]

paper_content:

The face conveys information about a person's age, sex, background, and identity; what they are feeling, thinking, or likely to do next. Facial expression regulates face-to-face interactions, indicates reciprocity and interpersonal attraction or repulsion, and enables intersubjectivity between members of different cultures. Facial expression indexes neurological and psychiatric functioning and reveals personality and socioemotional development. Not surprisingly, the face has been of keen interest to behavioral scientists. About 15 years ago, computer scientists became increasingly interested in the use of computer vision and graphics to automatically analyze and synthesize facial expression. This effort was made possible in part by the development in psychology of detailed coding systems for describing facial actions and their relation to basic emotions, that is, emotions that are interpreted similarly in diverse cultures. The most detailed of these systems, the Facial Action Coding System (FACS), informed the development of the MPEG-4 facial animation parameters for video transmission and enabled progress toward automated measurement and synthesis of facial actions for research in affective computing, social signal processing, and behavioral science. This article reports key advances in behavioral science that are becoming possible through these developments. Before beginning, automated facial image analysis and synthesis (AFAS) is briefly described.
---

Title: Interpersonal Synchrony: A Survey of Evaluation Methods across Disciplines

Section 1: Multidisciplinary Need for Synchrony Assessment
Description 1: Discuss the need for a multidisciplinary approach to assessing synchrony, its importance in various fields, and the challenges posed by the lack of automatic tools.

Section 2: Scope of this Paper
Description 2: Outline the main goals of the paper, including clarifying the concept of synchrony, reviewing evaluation methods, and presenting current limitations and future prospects.

Section 3: Related Theories and Definitions
Description 3: Define synchrony and related concepts, and provide an overview of relevant theories in the study of communicative interaction and language.

Section 4: Content Coordination
Description 4: Explain the coordination at the content level in conversation, including interactive alignment and its related features such as lexical, syntactic, and speech rate alignment.

Section 5: Process Coordination
Description 5: Discuss the temporal coordination between conversational partners and its role in achieving synchrony, as well as the criteria for its emergence.

Section 6: Proposed Definition of Synchrony
Description 6: Provide the authors' definition of synchrony, distinguishing it from related phenomena like mirroring or mimicry.

Section 7: Functions in Early Infancy
Description 7: Describe the roles and importance of synchrony during early development, especially in affective state regulation, social connection, attachment, language acquisition, and imitation.

Section 8: Functions in Adulthood
Description 8: Discuss the benefits of interactional synchrony in adulthood, such as facilitating social interactions, building rapport, cooperation, and its therapeutic implications.

Section 9: Noncomputational Methods of Synchrony Assessment
Description 9: Review traditional, noncomputational methods for evaluating synchrony, such as behavioral coding and judgment methods, including their advantages and limitations.

Section 10: Fully Automatic Measures of Movement Synchrony
Description 10: Explore computational techniques for measuring movement synchrony automatically, including the extraction of features, correlation, recurrence analysis, and spectral methods.

Section 11: Modeling Communication Dynamics
Description 11: Review models that describe the timing and occurrence of interactive behaviors and their application in understanding and predicting communication dynamics, such as turn-taking and backchannels.

Section 12: Open Questions
Description 12: Identify and discuss unresolved questions and challenges in the study of synchrony, including the time scale, dimensions, and the need for dedicated corpora.

Section 13: Prospects in Developmental Robotics
Description 13: Discuss the application of synchrony studies in developmental robotics for the autonomous development of skills, inspired by human-infant interactions.

Section 14: Prospects in Social Robots and Embodied Conversational Agents
Description 14: Examine how synchrony can enhance human-machine interaction, focusing on the creation of socially engaging and accepted interfaces in social robots and virtual agents.

Section 15: Prospects in Developmental and Clinical Studies
Description 15: Consider the application of synchrony assessments in clinical settings, particularly for children with ASD, and the potential benefits in therapeutic contexts.

Section 16: Conclusion
Description 16: Summarize the key findings and implications of the paper, emphasizing the reciprocal benefits between computational and psychological approaches to studying synchrony.