Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing
---
paper_title: Social Cognition in Humans

paper_content:

We review a diversity of studies of human social interaction and highlight the importance of social signals. We also discuss recent findings from social cognitive neuroscience that explore the brain basis of the capacity for processing social signals. These signals enable us to learn about the world from others, to learn about other people, and to create a shared social world. Social signals can be processed automatically by the receiver and may be unconsciously emitted by the sender. These signals are non-verbal and are responsible for social learning in the first year of life. Social signals can also be processed consciously and this allows automatic processing to be modulated and overruled. Evidence for this higher-level social processing is abundant from about 18 months of age in humans, while evidence is sparse for non-human animals. We suggest that deliberate social signalling requires reflective awareness of ourselves and awareness of the effect of the signals on others. Similarly, the appropriate reception of such signals depends on the ability to take another person's point of view. This ability is critical to reputation management, as this depends on monitoring how our own actions are perceived by others. We speculate that the development of these high level social signalling systems goes hand in hand with the development of consciousness.
---
paper_title: An Introduction to the Physiology of Hearing

paper_content:

This book deals with the way that the auditory system processes acoustic signals. The current edition has been thoroughly revised to reflect the progress that has been made since the previous edition. Particularly major updates have been made in the following areas: cochlear function, including cochlear mechanics, hair cell function and mechanisms of transduction; the auditory central nervous system, a major area of advance in recent years; physiological correlates of auditory perception, including speech perception; and, cochlear pathophysiology and sensorineural hearing loss, including the restoration of hearing by electrical stimulation of the ear, and molecular and cellular approaches to hair cell repair, replacement, and regeneration.A reading scheme has been provided to guide readers to the section most appropriate for their interests. The book is written so that those entering auditory research from very little background in auditory neuroscience are able to understand the current research issues and research literature. It is also intended to be a source book and reference work for advanced undergraduates studying the special senses, and for clinicians in the speciality of Otorhinolaryngology.It offers a contemporary look at the physiology of hearing: each chapter has been thoroughly revised. It is an excellent reading companion to practitioners and scholars. It is also suitable for those undertaking auditory research. It includes a reading scheme to guide readers through the book.
---
paper_title: Cognitive modelling of human social signals

paper_content:

The paper defines as "social signal" a communicative or informative signal that, either directly or indirectly, conveys information about social actions, social interactions, social emotions, social attitudes and social relationships. It proposes a conceptual definition of these social facts and some examples of relevant social signals.
---
paper_title: Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship

paper_content:

Interfaces that talk and listen are populating computers, cars, call centers, and even home appliances and toys, but voice interfaces invariably frustrate rather than help. In Wired for Speech, Clifford Nass and Scott Brave reveal how interactive voice technologies can readily and effectively tap into the automatic responses all speech -- whether from human or machine -- evokes. Wired for Speech demonstrates that people are "voice-activated": we respond to voice technologies as we respond to actual people and behave as we would in any social situation. By leveraging this powerful finding, voice interfaces can truly emerge as the next frontier for efficient, user-friendly technology.Wired for Speech presents new theories and experiments and applies them to critical issues concerning how people interact with technology-based voices. It considers how people respond to a female voice in e-commerce (does stereotyping matter?), how a car's voice can promote safer driving (are "happy" cars better cars?), whether synthetic voices have personality and emotion (is sounding like a person always good?), whether an automated call center should apologize when it cannot understand a spoken request ("To Err is Interface; To Blame, Complex"), and much more. Nass and Brave's deep understanding of both social science and design, drawn from ten years of research at Nass's Stanford laboratory, produces results that often challenge conventional wisdom and common design practices. These insights will help designers and marketers build better interfaces, scientists construct better theories, and everyone gain better understandings of the future of the machines that speak with us.
---
paper_title: Socially aware computation and communication

paper_content:

By building machines that understand social signaling and social context, we can dramatically improve collective decision making and help keep remote users 'in the loop.' I will describe three systems that have a substantial understanding of social context, and use this understanding to improve human group performance. The first system is able to interpret social displays of interest and attraction, and uses this information to improve conferences and meetings. The second is able to infer friendship, acquaitance, and workgroup relationships, and uses this to help people build social capital. The third is able to examine human interactions and categorize participants attitudes (attentive, agreeable, determined, interested, etc), and uses this information to proactively promote group cohesion and to match participants on the basis of their compatiblity.
---
paper_title: The mirror-neuron system

paper_content:

� Abstract A category of stimuli of great importance for primates, humans in particular, is that formed by actions done by other individuals. If we want to survive, we must understand the actions of others. Furthermore, without action understanding, social organization is impossible. In the case of humans, there is another faculty that depends on the observation of others’ actions: imitation learning. Unlike most species, we are able to learn by imitation, and this faculty is at the basis of human culture. In this review we present data on a neurophysiological mechanism—the mirror-neuron mechanism—that appears to play a fundamental role in both action understanding and imitation. We describe first the functional properties of mirror neurons in monkeys. We review next the characteristics of the mirror-neuron system in humans. We stress, in particular, those properties specific to the human mirror-neuron system that might explain the human capacity to learn by imitation. We conclude by discussing the relationship between the mirror-neuron system and language.
---
paper_title: Social Intelligence: The New Science of Success

paper_content:

Foreword. Acknowledgment. Preface. 1. A Different Kind of "Smart". Old Wine in New Bottles? Going Beyond IQ. EI, SI, or Both? From Toxic to Nourishing. Blind Spots, Lenses, and Filters. Social Halitosis, Flatulence, and Dandruff. The "Dilbert" Factor. Can We Become a Socially Smarter Species? S.P.A.C.E.:The Skills of Interaction. 2. "S" Stands for Situational Awareness. Situational Dumbness and Numbness. Ballistic Podiatry: Making the Worst of a Situation. Reading the Social Context. What to Look For. The Proxemic Context. The Behavioral Context. The Semantic Context. Navigating Cultures and Subcultures. Codes of Conduct: Violate the Rules at Your Peril. Building the Skills of Situational Awareness. 3. "P" Stands for Presence. Being There. Is Charisma Over-Rated? Do Looks Matter? Reading (and Shaping) the "Rules of Engagement". The Ugly American Syndrome. More of You, Less of Me. A Case of Attitude. Building the Skills of Presence. 4. "A" Stands for Authenticity. Take a Tip from Popeye. It's a Beautiful Day in the SI Neighborhood. The Snap-On Smile: Can You Fake Sincerity? Left-Handed Compliments. The Puppy Dog Syndrome. Narcissism: It's Really All About Me. Head Games, Power Struggles, and Manipulation. Building the Skills of Authenticity. 5. "C" Stands for Clarity. A Way with Words. Hoof-in-Mouth Disease: Sometimes Silence Works Best. Role-Speak and Real-Speak. Helicopter Language and Elevator Speeches. "Clean" Language and "Dirty" Language. Verbal Bludgeons. Taking a Brain for a Walk. The Power of Metaphor. E-Prime: the Language of Sanity. Building the Skills of Clarity. 6. "E" Stands for Empathy. What Destroys Empathy? What Builds Empathy? The Platinum Rule. The Irony of Empathic Professions. L.E.A.P.S.: Empathy by Design. Empathy in Four Minutes. Building the Skills of Empathy. 7. Assessing and Developing SI. Assessing Your Interaction Skills. Self-Awareness: Seeing Yourself as Others See You. Assessing Your Interaction Style: Drivers, Energizers, Diplomats, and Loners. The Strength-Weakness Irony. Priorities for Improvement. 8. SI in the World of Work: Some Reflections. The Real and Legal Consequences of Social Incompetence. Cultures of Conflict and Craziness. Hierarchies, Testosterone, and Gender Politics. Getting it Right at Work and Wrong at Home. The Diversity Puzzle. Ritual, Ceremony, and Celebration. Positive Politics: Getting Ahead with Your Value System Intact. 9. SI in Charge: Thoughts on Developing Socially Intelligent Leaders. The S.O.B. Factor. Executive Hubris: Its Costs and Consequences. Best Boss, Worst Boss. P.O.W.E.R.: Where It Comes From, How to Get It. How the Worst Bastards on the Planet Get and Keep Power. The Algebra of Influence. S.P.I.C.E.: Leading When You're Not In Charge. 10. SI and Conflict: Thoughts About Getting Along. The Double Spiral of Conflict. Why Argue? Crucial Conversations. Added Value Negotiating. Epilogue. SI and the Next Generation: Who's Teaching Our Kids? Our Children Are Not Our Children. The (Only) Ten Basic News Stories. Anxiety Drives Attention. Breaking the Addiction to Television. The Buying of Our Babies. Video Games:The New Sandlot. Teachers, Parents, or Neither? Belonging or Be Longing? The S.P.A.C.E. Solution for Schools. A Prescription for SI at Every Age. Index. About the Author.
---
paper_title: Social Computing: From Social Informatics to Social Intelligence

paper_content:

Social computing represents a new computing paradigm and an interdisciplinary research and application field. Undoubtedly, it strongly influences system and software developments in the years to come. We expect that social computing's scope continues to expand and its applications multiply. From both theoretical and technological perspectives, social computing technologies moves beyond social information processing towards emphasizing social intelligence. As we've discussed, the move from social informatics to social intelligence is achieved by modeling and analyzing social behavior, by capturing human social dynamics, and by creating artificial social agents and generating and managing actionable social knowledge
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: Selection for universal facial emotion

paper_content:

Facial expression is heralded as a communication system common to all human populations, and thus is generally accepted as a biologically based, universal behavior. Happiness, sadness, fear, anger, surprise, and disgust are universally recognized and produced emotions, and communication of these states is deemed essential in order to navigate the social environment. It is puzzling, however, how individuals are capable of producing similar facial expressions when facial musculature is known to vary greatly among individuals. Here, the authors show that although some facial muscles are not present in all individuals, and often exhibit great asymmetry (larger or absent on one side), the facial muscles that are essential in order to produce the universal facial expressions exhibited 100% occurrence and showed minimal gross asymmetry in 18 cadavers. This explains how universal facial expression production is achieved, implies that facial muscles have been selected for essential nonverbal communicative function, and yet also accommodate individual variation.
---
paper_title: Social signal processing: What are the relevant variables? And in what ways do they relate?

paper_content:

Studies of the processing of social signals and behaviour tend to focus intuitively on a few variables, without a framework to guide selection. Here, we attempt to provide a broad overview of the relevant variables, describing both signs and what they signify. Those are matched by systematic consideration of how the variables relate. Variables interact not only on an intrapersonal level but also on an interpersonal level. It is also recognised explicitly that a comprehensive framework needs to embrace the role of context and individual differences in personality and culture.
---
paper_title: Crossmodal binding of fear in voice and face

paper_content:

In social environments, multiple sensory channels are simultaneously engaged in the service of communication. In this experiment, we were concerned with defining the neuronal mechanisms for a perceptual bias in processing simultaneously presented emotional voices and faces. Specifically, we were interested in how bimodal presentation of a fearful voice facilitates recognition of fearful facial expression. By using event-related functional MRI, that crossed sensory modality (visual or auditory) with emotional expression (fearful or happy), we show that perceptual facilitation during face fear processing is expressed through modulation of neuronal responses in the amygdala and the fusiform cortex. These data suggest that the amygdala is important for emotional crossmodal sensory convergence with the associated perceptual bias during fear processing, being mediated by task-related modulation of face-processing regions of fusiform cortex.
---
paper_title: Integrating Face and Voice in Person Perception

paper_content:

Integration of information from face and voice plays a central role in our social interactions. It has been mostly studied in the context of audiovisual speech perception: integration of affective or identity information has received comparatively little scientific attention. Here, we review behavioural and neuroimaging studies of face-voice integration in the context of person perception. Clear evidence for interference between facial and vocal information has been observed during affect recognition or identity processing. Integration effects on cerebral activity are apparent both at the level of heteromodal cortical regions of convergence, particularly bilateral posterior superior temporal sulcus (pSTS), and at 'unimodal' levels of sensory processing. Whether the latter reflects feedback mechanisms or direct crosstalk between auditory and visual cortices is as yet unclear.
---
paper_title: Nonverbal Behavior and Nonverbal Communication: What do Conversational Hand Gestures Tell Us?

paper_content:

Publisher Summary This chapter explores how gestures contribute to comprehension, how gesturing affect speech and what can be learned from studying conversational gestures. The primary function of conversational hand gestures is to aid in the formulation of speech. Gestures can convey nonsemantic information. The study of speech and gestures overlaps with the study of person perception and attribution processes. The significance of gestures can be ambiguous and will affect the meanings and consequences to the observed gestures. A topology of gestures is adopters, symbolic gestures, and conversational gestures. Different types of conversational gestures can be distinguished as—namely, motor movements and lexical movements. Conversational hand gestures have been assumed to convey semantic information. Several studies that attempt to assess the kinds of information conversational gestures convey to naive observers and the extent to which gestures enhance the communicativeness of spoken messages are described in the chapter.
---
paper_title: The Effects of the Gesture Viewpoint on the Students' Memory of Words and Stories

paper_content:

The goal of this work is to estimate the effects of teacher's iconic gestures on the students' memory of words and short stories. Indeed, some evidence seems to confirm the possibility that iconics help the listener, but it is unclear what are the elements that make gestures more or less useful. According to McNeill's observation that children produce many more Character Viewpoint gestures than Observer Viewpoint ones, we hypothesize that they also understand and remember better words accompanied by these gestures. The results of two experimental studies showed that iconic gestures helped students to remember words and tales better and that younger students performed better in memory tasks when their teacher used Character Viewpoint gestures.
---
paper_title: Implementing Expressive Gesture Synthesis for Embodied Conversational Agents

paper_content:

We aim at creating an expressive Embodied Conversational Agent (ECA) and address the problem of synthesizing expressive agent gestures. In our previous work, we have described the gesture selection process. In this paper, we present a computational model of gesture quality. Once a certain gesture has been chosen for execution, how can we modify it to carry a desired expressive content while retaining its original semantics? We characterize bodily expressivity with a small set of dimensions derived from a review of psychology literature. We provide a detailed description of the implementation of these dimensions in our animation system, including our gesture modeling language. We also demonstrate animations with different expressivity settings in our existing ECA system. Finally, we describe two user studies that evaluate the appropriateness of our implementation for each dimension of expressivity as well as the potential of combining these dimensions to create expressive gestures that reflect communicative intent.
---
paper_title: Communicative Feedback Phenomena across Languages and Modalities

paper_content:

This research deals with human communicative behaviour related to feedback, analysed across languages (Italian and Swedish), modalities (auditory versus visual) and different communicative situations (human-human versus human-machine dialogues). The aim is to give more insight into how humans express feedback and at the same time suggest a method to collect valuable data that can be useful to control facial and head movements related to visual feedback in synthetic conversational agents. The analysed data span from spontaneous conversations video-recorded in real communicative situations, and semi spontaneous dialogues obtained with different eliciting techniques, to a specific corpus of controlled interactive speech collected by means of a motion capture system. A specific coding scheme has been developed, tested and used to annotate feedback with the support of different available software packages for audio visual analysis. This study should be especially useful to professionals in Speech Communication and Speech Technology, but even Psychologists studying human behavior in human-human and human-machine interactions might find it interesting.
---
paper_title: Speaking: From Intention to Articulation

paper_content:

In Speaking, Willem "Pim" Levelt, Director of the Max-Planck-Institut fur Psycholinguistik, accomplishes the formidable task of covering the entire process of speech production, from constraints on conversational appropriateness to articulation and self-monitoring of speech. Speaking is unique in its balanced coverage of all major aspects of the production of speech, in the completeness of its treatment of the entire speech process, and in its strategy of exemplifying rather than formalizing theoretical issues.
---
paper_title: Honest Signals: How They Shape Our World

paper_content:

How can you know when someone is bluffing? Paying attention? Genuinely interested? The answer, writes Sandy Pentland in Honest Signals, is that subtle patterns in how we interact with other people reveal our attitudes toward them. These unconscious social signals are not just a back channel or a complement to our conscious language; they form a separate communication network. Biologically based "honest signaling," evolved from ancient primate signaling mechanisms, offers an unmatched window into our intentions, goals, and values. If we understand this ancient channel of communication, Pentland claims, we can accurately predict the outcomes of situations ranging from job interviews to first dates. Pentland, an MIT professor, has used a specially designed digital sensor worn like an ID badgea "sociometer"to monitor and analyze the back-and-forth patterns of signaling among groups of people. He and his researchers found that this second channel of communication, revolving not around words but around social relations, profoundly influences major decisions in our liveseven though we are largely unaware of it. Pentland presents the scientific background necessary for understanding this form of communication, applies it to examples of group behavior in real organizations, and shows how by "reading" our social networks we can become more successful at pitching an idea, getting a job, or closing a deal. Using this "network intelligence" theory of social signaling, Pentland describes how we can harness the intelligence of our social network to become better managers, workers, and communicators.
---
paper_title: Laws of organization in perceptual forms.

paper_content:

Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees. It is impossible to achieve "327 " as such. And yet even though such droll calculation were possible and implied, say, for the house 120, the trees 90, the sky 117 -I should at least have this arrangement and division of the total, and not, say, 127 and 100 and 100; or 150 and 177.
---
paper_title: Challenges ahead head movements and other social acts in conversation

paper_content:

When involved in face-to-face conversations, people move their heads in typical ways. The pattern of head gestures and their function in conversation has been studied in various disciplines. Many factors are involved in determining the exact patterns that occur in conversation. These can be explained by considering some of the basic properties of face-to-face interactions. The fact that conversations are a type of joint activity involving social actions together with a few other properties, such as the need for grounding, can explain the variety in functions that are served by the multitude of movements that people display during conversations.
---
paper_title: MULTIMODALITY IN OWN COMMUNICATION MANAGEMENT

paper_content:

This study studies how gestures (here defined in a wide sense, including all body movements which have a communicative function) are used for Own Communication Management (OCM), an interesting and not completely well described part of the language system. OCM concerns how a speaker continuously manages the planning and execution of his/her own communication and is a basic function in face-to-face interaction. It has two main functions, i.e. "choice" and "change". The study investigates how much of OCM involves gestures and whether there is a difference between choice and change OCM in this respect. It also concerns what kinds of gestures are used in OCM and what the relation is between vocal and gestural OCM. Some of the main findings are that roughly 50% of all speech based OCMcooccurs with gestures and that most of the OCM involving gestures (about 90%) is choice directed. Gestures occurring with OCM can illustrate the content of a sought after word, but also more generally induce word activation. They can also signal to an interlocutor that the speaker needs time. Gestures are often multifunctional and, thus, both choice and change are often integrated with more interactive functions. A final observation is that gestural OCM either precedes or occurs simultaneously with verbal OCM.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: Communicative Feedback Phenomena across Languages and Modalities

paper_content:

This research deals with human communicative behaviour related to feedback, analysed across languages (Italian and Swedish), modalities (auditory versus visual) and different communicative situations (human-human versus human-machine dialogues). The aim is to give more insight into how humans express feedback and at the same time suggest a method to collect valuable data that can be useful to control facial and head movements related to visual feedback in synthetic conversational agents. The analysed data span from spontaneous conversations video-recorded in real communicative situations, and semi spontaneous dialogues obtained with different eliciting techniques, to a specific corpus of controlled interactive speech collected by means of a motion capture system. A specific coding scheme has been developed, tested and used to annotate feedback with the support of different available software packages for audio visual analysis. This study should be especially useful to professionals in Speech Communication and Speech Technology, but even Psychologists studying human behavior in human-human and human-machine interactions might find it interesting.
---
paper_title: Perception of non-verbal emotional listener feedback

paper_content:

This paper reports on a listening test assessing the perception of short non-verbal emotional vocalisations emitted by a listener as feedback to the speaker. We clarify the concepts backchannel and feedback, and investigate the use of affect bursts as a means of giving emotional feedback via the backchannel. Experiments with German and Dutch subjects confirm that the recognition of emotion from affect bursts in a dialogical context is similar to their perception in isolation. We also investigate the acceptability of affect bursts when used as listener feedback. Acceptability appears to be linked to display rules for emotion expression. While many ratings were similar between Dutch and German listeners, a number of clear differences was found, suggesting language-specific affect bursts.
---
paper_title: Social signals and the action — Cognition loop. The case of overhelp and evaluation

paper_content:

The paper explores the action — cognition loop by investigating the relation between overhelp and evaluation. It presents a study on the helping and overhelping behaviors of teachers with students of their own vs. of a stigmatized culture, and analyses them in terms of a taxonomy of helping behavior, and adopting an annotation scheme to assess the multimodal behavior of teachers and pupils. Results show that overhelping teachers induce more negative evaluations, more often concerning general capacities, and frequently expressed indirectly. This seems to show that the overhelp offered blocks a child's striving for autonomy since it generates a negative evaluation, in particular the belief of an inability of the receiver.
---
paper_title: GraphLaugh: A tool for the interactive generation of humorous puns

paper_content:

While automatic generation of funny texts delivers incrementally better results, for the time being semiautomatic generation can already provide something useful. In particular, we present an interactive system for producing humorous puns obtained through variation (i.e., word substitution) performed on familiar expressions. The replacement word is selected according to phonetic similarity and semantic constraints expressing semantic opposition or evoking ridiculous traits of people. Examples of these puns are Chaste makes waste (variation on proverb) and Genital Hospital (variation on soap opera title). Lexical substitution is the humorous core on which funniness of pun is based. We implemented an interactive tool (called GraphLaugh) that can automatically generate different type of lexical associations and visualize them through a dynamic graph. Through the interaction with the network nodes and arcs, the user can control the selection of words, semantic associations and familiar expressions. In this way, a restrict set of familiar expressions are filtered, the best word substitutions to apply them are easily individuated, and finally a list of funny puns is created.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: Spotting agreement and disagreement: A survey of nonverbal audiovisual cues and tools

paper_content:

While detecting and interpreting temporal patterns of non-verbal behavioral cues in a given context is a natural and often unconscious process for humans, it remains a rather difficult task for computer systems. Nevertheless, it is an important one to achieve if the goal is to realise a naturalistic communication between humans and machines. Machines that are able to sense social attitudes like agreement and disagreement and respond to them in a meaningful way are likely to be welcomed by users due to the more natural, efficient and human-centered interaction they are bound to experience. This paper surveys the nonverbal cues that could be present during agreement and disagreement behavioural displays and lists a number of tools that could be useful in detecting them, as well as a few publicly available databases that could be used to train these tools for analysis of spontaneous, audiovisual instances of agreement and disagreement.
---
paper_title: Nonverbal Behaviors, Persuasion, and Credibility

paper_content:

This study examined the relationships among nonverbal behaviors, dimensions of source credibility, and speaker persuasiveness in a public speaking context. Relevant nonverbal literature was organized according to a Brunswikian lens model. Nonverbal behavioral composites, grouped according to their likely proximal percepts, were hypothesized to significantly affect both credibility and persuasiveness. A sample of 60 speakers gave videotaped speeches that were judged on credibility and persuasiveness by classmates. Pairs of trained raters coded 22 vocalic, kinesic, and proxemic nonverbal behaviors evidenced in the tapes. Results confirmed numerous associations between nonverbal behaviors and attributions of credibility and persuasiveness. Greater perceived competence and composure were associated with greater vocal and facial pleasantness, with greater facial expressiveness contributing to competence perceptions. Greater sociability was associated with more kinesic/proxemic immediacy, dominance, and relaxation and with vocal pleasantness. Most of these same cues also enhanced character judgments. No cues were related to dynamism judgments. Greater perceived persuasiveness correlated with greater vocal pleasantness (especially fluency and pitch variety), kinesic/proxemic immediacy, facial expressiveness, and kinesic relaxation (especially high random movement but little tension). All five dimensions of credibility related to persuasiveness. Advantages of analyzing nonverbal cues according to proximal percepts are discussed.
---
paper_title: Behavioral markers and recognizability of the smile of enjoyment.

paper_content:

Ekman and Friesen (1982) predicted that smiles that express enjoyment would be marked by smoother zygomatic major actions of more consistent duration than the zygomatic major actions of nonenjoyment smiles. Study 1 measured the duration and smoothness of smiles shown by female subjects in response to positive emotion films while alone and in a social interaction. Enjoyment smiles in both situations were of more consistent duration and smoother than nonenjoyment smiles. In Study 2 observers who were shown videotapes of enjoyment and nonenjoyment smiles were able to accurately identify enjoyment smiles at rates greater than chance; moreover, accuracy was positively related to increased salience of orbicularis oculi action. In Study 3, another group of observers were asked to record their impressions of the smiling women shown in Study 2. These women were seen as more positive when they showed enjoyment compared with nonenjoyment smiles. These results provide further evidence that enjoyment smiles are entities distinct from smiles in general.
---
paper_title: The signs of appeasement: Evidence for the distinct displays of embarrassment, amusement, and shame

paper_content:

According to appeasement hypotheses, embarrassment should have a distinct nonverbal display that is more readily perceived when displayed by individuals from lower status groups. The evidence from 5 studies supported these two claims. The nonverbal behavior of embarrassment was distinct from a related emotion (amusement), resembled the temporal pattern of facial expressions of emotion, was uniquely related to self-reports of embarrassment, and was accurately identified by observers who judged the spontaneous displays of various emotions. Across the judgment studies, observers were more accurate and attributed more emotion to the embarrassment displays of female and AfricanAmerican targets than those of male and Caucasian targets. Discussion focused on the universality and appeasement function of the embarrassment display. Since universal facial expressions of a limited set of emotions were first documented (Ekman & Friesen, 1971; Ekman, Sorenson, & Friesen, 1969; Izard, 1971), sparse attention has been given to facial expressions of other emotions. The resulting lacuna in the field—that the emotions with identified displays are fewer (7 to 10) than the states that lay people (Fehr & Russell, 1984) and emotion theorists (Ekman, 1992; Izard, 1977; Tomkins, 1963, 1984) label as emotions—presents intriguing possibilities. Displays of other emotions may be blends of other emotional displays, unidentifiable, or may await discovery.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: An overview of automatic speaker diarization systems

paper_content:

Audio diarization is the process of annotating an input audio channel with information that attributes (possibly overlapping) temporal regions of signal energy to their specific sources. These sources can include particular speakers, music, background noise sources, and other signal source/channel characteristics. Diarization can be used for helping speech recognition, facilitating the searching and indexing of audio archives, and increasing the richness of automatic transcriptions, making them more readable. In this paper, we provide an overview of the approaches currently used in a key area of audio diarization, namely speaker diarization, and discuss their relative merits and limitations. Performances using the different techniques are compared within the framework of the speaker diarization task in the DARPA EARS Rich Transcription evaluations. We also look at how the techniques are being introduced into real broadcast news systems and their portability to other domains and tasks such as meetings and speaker verification
---
paper_title: Gesture Recognition: A Survey

paper_content:

Gesture recognition pertains to recognizing meaningful expressions of motion by a human, involving the hands, arms, face, head, and/or body. It is of utmost importance in designing an intelligent and efficient human-computer interface. The applications of gesture recognition are manifold, ranging from sign language through medical rehabilitation to virtual reality. In this paper, we provide a survey on gesture recognition with particular emphasis on hand gestures and facial expressions. Applications involving hidden Markov models, particle filtering and condensation, finite-state machines, optical flow, skin color, and connectionist models are discussed in detail. Existing challenges and future research possibilities are also highlighted
---
paper_title: A survey of affect recognition methods: audio, visual and spontaneous expressions

paper_content:

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. Promising approaches have been reported, including automatic methods for facial and vocal affect recognition. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions-despite the fact that deliberate behavior differs in visual and audio expressions from spontaneously occurring behavior. Recently efforts to develop algorithms that can process naturally occurring human affective behavior have emerged. This paper surveys these efforts. We first discuss human emotion perception from a psychological perspective. Next, we examine the available approaches to solving the problem of machine understanding of human affective behavior occurring in real-world settings. We finally outline some scientific and engineering challenges for advancing human affect sensing technology.
---
paper_title: Detecting Faces in Images : A Survey

paper_content:

Images containing faces are essential to intelligent vision-based human-computer interaction, and research efforts in face processing include face recognition, face tracking, pose estimation and expression recognition. However, many reported methods assume that the faces in an image or an image sequence have been identified and localized. To build fully automated systems that analyze the information contained in face images, robust and efficient face detection algorithms are required. Given a single image, the goal of face detection is to identify all image regions which contain a face, regardless of its 3D position, orientation and lighting conditions. Such a problem is challenging because faces are non-rigid and have a high degree of variability in size, shape, color and texture. Numerous techniques have been developed to detect faces in a single image, and the purpose of this paper is to categorize and evaluate these algorithms. We also discuss relevant issues such as data collection, evaluation metrics and benchmarking. After analyzing these algorithms and identifying their limitations, we conclude with several promising directions for future research.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: D.: Computational Studies of Human Motion: Part 1, Tracking and Motion Synthesis

paper_content:

We review methods for kinematic tracking of the human body in video. The review is part of a projected book that is intended to cross-fertilize ideas about motion representation between the animation and computer vision communities. The review confines itself to the earlier stages of motion, focusing on tracking and motion synthesis; future material will cover activity representation and motion generation.
---
paper_title: SMaRT: the Smart Meeting Room Task at ISL

paper_content:

As computational and communications systems become increasingly smaller, faster, more powerful, and more integrated, the goal of interactive, integrated meeting support rooms is slowly becoming reality. It is already possible, for instance, to rapidly locate task-related information during a meeting, filter it, and share it with remote users. Unfortunately, the technologies that provide such capabilities are as obstructive as they are useful - they force humans to focus on the tool rather than the task. Thus the veneer of utility often hides the true costs of use, which are longer, less focused human interactions. To address this issue, we present our current research efforts towards SMaRT: the Smart Meeting Room Task. The goal of SMaRT is to provide meeting support services that do not require explicit human-computer interaction. Instead, by monitoring the activities in the meeting room using both video and audio analysis, the room is able to react appropriately to users' needs and allow the users to focus on their own goals.
---
paper_title: Modeling human interaction in meetings

paper_content:

The paper investigates the recognition of group actions in meetings by modeling the joint behaviour of participants. Many meeting actions, such as presentations, discussions and consensus, are characterised by similar or complementary behaviour across participants. Recognising these meaningful actions is an important step towards the goal of providing effective browsing and summarisation of processed meetings. A corpus of meetings was collected in a room equipped with a number of microphones and cameras. The corpus was labeled in terms of a predefined set of meeting actions characterised by global behaviour. In experiments, audio and visual features for each participant are extracted from the raw data and the interaction of participants is modeled using HMM-based approaches. Initial results on the corpus demonstrate the ability of the system to recognise the set of meeting actions.
---
paper_title: Automatic nonverbal analysis of social interaction in small groups : A review

paper_content:

An increasing awareness of the scientific and technological value of the automatic understanding of face-to-face social interaction has motivated in the past few years a surge of interest in the devising of computational techniques for conversational analysis. As an alternative to existing linguistic approaches for the automatic analysis of conversations, a relatively recent domain is using findings in social cognition, social psychology, and communication that have established the key role that nonverbal communication plays in the formation, maintenance, and evolution of a number of fundamental social constructs, which emerge from face-to-face interactions in time scales that range from short glimpses all the way to long-term encounters. Small group conversations are a specific case on which much of this work has been conducted. This paper reviews the existing literature on automatic analysis of small group conversations using nonverbal communication, and aims at bridging the current fragmentation of the work in this domain, currently split among half a dozen technical communities. The review is organized around the main themes studied in the literature and discusses, in a comparative fashion, about 100 works addressing problems related to the computational modeling of interaction management, internal states, personality traits, and social relationships in small group conversations, along with pointers to the relevant literature in social science. Some of the many open challenges and opportunities in this domain are also discussed.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: C4.5: Programs for Machine Learning

paper_content:

From the Publisher: ::: Classifier systems play a major role in machine learning and knowledge-based systems, and Ross Quinlan's work on ID3 and C4.5 is widely acknowledged to have made some of the most significant contributions to their development. This book is a complete guide to the C4.5 system as implemented in C for the UNIX environment. It contains a comprehensive guide to the system's use , the source code (about 8,800 lines), and implementation notes. The source code and sample datasets are also available on a 3.5-inch floppy diskette for a Sun workstation. ::: ::: C4.5 starts with large sets of cases belonging to known classes. The cases, described by any mixture of nominal and numeric properties, are scrutinized for patterns that allow the classes to be reliably discriminated. These patterns are then expressed as models, in the form of decision trees or sets of if-then rules, that can be used to classify new cases, with emphasis on making the models understandable as well as accurate. The system has been applied successfully to tasks involving tens of thousands of cases described by hundreds of properties. The book starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting. Advantages and disadvantages of the C4.5 approach are discussed and illustrated with several case studies. ::: ::: This book and software should be of interest to developers of classification-based intelligent systems and to students in machine learning and expert systems courses.
---
paper_title: The Rules Behind Roles: Identifying Speaker Role in Radio Broadcasts

paper_content:

Previous work has shown that providing information about story structure is critical for browsing audio broadcasts. We investigate the hypothesis that Speaker Role is an important cue to story structure. We implement an algorithm that classies story segments into three Speaker Roles based on several content and duration features. The algorithm correctly classies about 80% of segments (compared with a baseline frequency of 35.4%) when applied to ASR derived transcriptions of broadcast data.
---
paper_title: Speakers Role Recognition in Multiparty Audio Recordings Using Social Network Analysis and Duration Distribution Modeling

paper_content:

This paper presents two approaches for speaker role recognition in multiparty audio recordings. The experiments are performed over a corpus of 96 radio bulletins corresponding to roughly 19 h of material. Each recording involves, on average, 11 speakers playing one among six roles belonging to a predefined set. Both proposed approaches start by segmenting automatically the recordings into single speaker segments, but perform role recognition using different techniques. The first approach is based on Social Network Analysis, the second relies on the intervention duration distribution across different speakers. The two approaches are used separately and combined and the results show that around 85% of the recording time can be labeled correctly in terms of role.
---
paper_title: Initial Study On Automatic Identification Of Speaker Role In Broadcast News Speech

paper_content:

Identifying a speaker's role (anchor, reporter, or guest speaker) is important for finding the structural information in broadcast news speech. We present an HMM-based approach and a maximum entropy model for speaker role labeling using Mandarin broadcast news speech. The algorithms achieve classification accuracy of about 80% (compared to the baseline of around 50%) using the human transcriptions and manually labeled speaker turns. We found that the maximum entropy model performs slightly better than the HMM, and that the combination of them outperforms any model alone. The impact of the contextual role information is also examined in this study.
---
paper_title: Automatic role recognition in multiparty recordings using social networks and probabilistic sequential models

paper_content:

The automatic analysis of social interactions is attracting significant interest in the multimedia community. This work addresses one of the most important aspects of the problem, namely the recognition of roles in social exchanges. The proposed approach is based on Social Network Analysis, for the representation of individuals in terms of their interactions with others, and probabilistic sequential models, for the recognition of role sequences underlying the sequence of speakers in conversations. The experiments are performed over different kinds of data (around 90 hours of broadcast data and meetings), and show that the performance depends on how formal the roles are, i.e. on how much they constrain people behavior.
---
paper_title: Using Simple Speech–Based Features to Detect the State of a Meeting and the Roles of the Meeting Participants

paper_content:

We introduce a simple taxonomy of meeting states and participant roles. Our goal is to automatically detect the state of a meeting and the role of each meeting participant and to do so concurrent with a meeting. We trained a decision tree classifier that learns to detect these states and roles from simple speech–based features that are easy to compute automatically. This classifier detects meeting states 18% absolute more accurately than a random classifier, and detects participant roles 10% absolute more accurately than a majority classifier. The results imply that simple, easy to compute features can be used for this purpose.
---
paper_title: Modeling Vocal Interaction for Text-Independent Participant Characterization in Multi-Party Conversation

paper_content:

An important task in automatic conversation understanding is the inference of social structure governing participant behavior. We explore the dependence between several social dimensions, including assigned role, gender, and seniority, and a set of low-level features descriptive of talkspurt deployment in a multiparticipant context. Experiments conducted on two large, publicly available meeting corpora suggest that our features are quite useful in predicting these dimensions, excepting gender. The classification experiments we present exhibit a relative error rate reduction of 37% to 67% compared to choosing the majority class.
---
paper_title: Automatic detection of group functional roles in face to face interactions

paper_content:

In this paper, we discuss a machine learning approach to automatically detect functional roles played by participants in a face to face interaction. We shortly introduce the coding scheme we used to classify the roles of the group members and the corpus we collected to assess the coding scheme reliability as well as to train statistical systems for automatic recognition of roles. We then discuss a machine learning approach based on multi-class SVM to automatically detect such roles by employing simple features of the visual and acoustical scene. The effectiveness of the classification is better than the chosen baselines and although the results are not yet good enough for a real application, they demonstrate the feasibility of the task of detecting group functional roles in face to face interactions.
---
paper_title: Automatic Role Recognition in Multiparty Recordings: Using Social Affiliation Networks for Feature Extraction

paper_content:

Automatic analysis of social interactions attracts increasing attention in the multimedia community. This letter considers one of the most important aspects of the problem, namely the roles played by individuals interacting in different settings. In particular, this work proposes an automatic approach for the recognition of roles in both production environment contexts (e.g., news and talk-shows) and spontaneous situations (e.g., meetings). The experiments are performed over roughly 90 h of material (one of the largest databases used for role recognition in the literature) and show that the recognition effectiveness depends on how much the roles influence the behavior of people. Furthermore, this work proposes the first approach for modeling mutual dependences between roles and assesses its effect on role recognition performance.
---
paper_title: Using the influence model to recognize functional roles in meetings

paper_content:

In this paper, an influence model is used to recognize functional roles played during meetings. Previous works on the same corpus demonstrated a high recognition accuracy using SVMs with RBF kernels. In this paper, we discuss the problems of that approach, mainly over-fitting, the curse of dimensionality and the inability to generalize to different group configurations. We present results obtained with an influence modeling method that avoid these problems and ensures both greater robustness and generalization capability.
---
paper_title: Role recognition for meeting participants: an approach based on lexical information and social network analysis

paper_content:

This paper presents experiments on the automatic recognition of roles in meetings. The proposed approach combines two sources of information: the lexical choices made by people playing different roles on one hand, and the Social Networks describing the interactions between the meeting participants on the other hand. Both sources lead to role recognition results significantly higher than chance when used separately, but the best results are obtained with their combination. Preliminary experiments obtained over a corpus of 138 meeting recordings (over 45 hours of material) show that around 70% of the time is labeled correctly in terms of role.
---
paper_title: Opinion mining and sentiment analysis

paper_content:

An important part of our information-gathering behavior has always been to find out what other people think. With the growing availability and popularity of opinion-rich resources such as online review sites and personal blogs, new opportunities and challenges arise as people now can, and do, actively use information technologies to seek out and understand the opinions of others. The sudden eruption of activity in the area of opinion mining and sentiment analysis, which deals with the computational treatment of opinion, sentiment, and subjectivity in text, has thus occurred at least in part as a direct response to the surge of interest in new systems that deal directly with opinions as a first-class object. ::: ::: This survey covers techniques and approaches that promise to directly enable opinion-oriented information-seeking systems. Our focus is on methods that seek to address the new challenges raised by sentiment-aware applications, as compared to those that are already present in more traditional fact-based analysis. We include material on summarization of evaluative text and on broader issues regarding privacy, manipulation, and economic impact that the development of opinion-oriented information-access services gives rise to. To facilitate future work, a discussion of available resources, benchmark datasets, and evaluation campaigns is also provided.
---
paper_title: A survey of affect recognition methods: audio, visual and spontaneous expressions

paper_content:

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. Promising approaches have been reported, including automatic methods for facial and vocal affect recognition. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions-despite the fact that deliberate behavior differs in visual and audio expressions from spontaneously occurring behavior. Recently efforts to develop algorithms that can process naturally occurring human affective behavior have emerged. This paper surveys these efforts. We first discuss human emotion perception from a psychological perspective. Next, we examine the available approaches to solving the problem of machine understanding of human affective behavior occurring in real-world settings. We finally outline some scientific and engineering challenges for advancing human affect sensing technology.
---
paper_title: Head Gestures for Perceptual Interfaces: The Role of Context in Improving Recognition

paper_content:

Head pose and gesture offer several conversational grounding cues and are used extensively in face-to-face interaction among people. To accurately recognize visual feedback, humans often use contextual knowledge from previous and current events to anticipate when feedback is most likely to occur. In this paper we describe how contextual information can be used to predict visual feedback and improve recognition of head gestures in human-computer interfaces. Lexical, prosodic, timing, and gesture features can be used to predict a user's visual feedback during conversational dialog with a robotic or virtual agent. In non-conversational interfaces, context features based on user-interface system events can improve detection of head gestures for dialog box confirmation or document browsing. Our user study with prototype gesture-based components indicate quantitative and qualitative benefits of gesture-based confirmation over conventional alternatives. Using a discriminative approach to contextual prediction and multi-modal integration, performance of head gesture detection was improved with context features even when the topic of the test set was significantly different than the training set.
---
paper_title: Identifying Agreement And Disagreement In Conversational Speech: Use Of Bayesian Networks To Model Pragmatic Dependencies

paper_content:

We describe a statistical approach for modeling agreements and disagreements in conversational interaction. Our approach first identifies adjacency pairs using maximum entropy ranking based on a set of lexical, durational, and structural features that look both forward and backward in the discourse. We then classify utterances as agreement or disagreement using these adjacency pairs and features that represent various pragmatic influences of previous agreement or disagreement on the current utterance. Our approach achieves 86.9% accuracy, a 4.9% increase over previous work.
---
paper_title: The Chameleon Effect as Social Glue: Evidence for the Evolutionary Significance of Nonconscious Mimicry

paper_content:

The "chameleon effect" refers to the tendency to adopt the postures, gestures, and mannerisms of interaction partners (Chartrand & Bargh, 1999). This type of mimicry occurs outside of conscious awareness, and without any intent to mimic or imitate. Empirical evidence suggests a bi-directional relationship between nonconscious mimicry on the one hand, and liking, rapport, and affiliation on the other. That is, nonconscious mimicry creates affiliation, and affiliation can be ex- pressed through nonconscious mimicry. We argue that mimicry played an impor- tant role in human evolution. Initially, mimicry may have had survival value by helping humans communicate. We propose that the purpose of mimicry has now evolved to serve a social function. Nonconscious behavioral mimicry increases af- filiation, which serves to foster relationships with others. We review current re- search in light of this proposed framework and suggest future areas of research.
---
paper_title: Dominance Detection in Meetings Using Easily Obtainable Features

paper_content:

We show that, using a Support Vector Machine classifier, it is possible to determine with a 75% success rate who dominated a particular meeting on the basis of a few basic features. We discuss the corpus we have used, the way we had people judge dominance and the features that were used.
---
paper_title: The Five-Factor Model of Personality

paper_content:

Study I in this chapter reports an Indian (Marathi) adaptation of the Revised NEO Personality Inventory (NEO-PI-R), its psychometric evaluation, and gender differences based on data from 214 subjects. Factor analyses supported the Five-Factor Model and indicated factorial invariance across Indian and American cultures. The study also demonstrated the utility of oblique and orthogonal Procrustes rotations and multiple group factor analysis in evaluating the Five-Factor Model. Study II, employing 300 subjects, examined the Eysenckian correlates of the Indian (Marathi) NEO Five-Factor Inventory (NEO-FFI). The obtained correlations provide validity evidence for the NEO-FFI and its parent instrument, the NEO-PI-R.
---
paper_title: A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances

paper_content:

A novel probabilistic framework is proposed for inferring the structure of conversation in face-to-face multiparty communication, based on gaze patterns, head directions and the presence/absence of utterances. As the structure of conversation, this study focuses on the combination of participants and their participation roles. First, we assess the gaze patterns that frequently appear in conversations, and define typical types of conversation structure, called conversational regime, and hypothesize that the regime represents the high-level process that governs how people interact during conversations. Next, assuming that the regime changes over time exhibit Markov properties, we propose a probabilistic conversation model based on Markov-switching; the regime controls the dynamics of utterances and gaze patterns, which stochastically yield measurable head-direction changes. Furthermore, a Gibbs sampler is used to realize the Bayesian estimation of regime, gaze pattern, and model parameters from observed head directions and utterances. Experiments on four-person conversations confirm the effectiveness of the framework in identifying conversation structures.
---
paper_title: Capturing Order in Social Interactions 1

paper_content:

Such an ambitious plan of filling the social intelligence gap between humans and machines starts from a fundamental problem, namely how to make social phenomena accessible to computers when the only evidence these have at disposition about the world are signals captured with devices like microphones and cameras. The consequent question is “do social phenomena leave physical, machine detectable, traces in signals
---
paper_title: Multimodal support to group dynamics

paper_content:

The complexity of group dynamics occurring in small group interactions often hinders the performance of teams. The availability of rich multimodal information about what is going on during the meeting makes it possible to explore the possibility of providing support to dysfunctional teams from facilitation to training sessions addressing both the individuals and the group as a whole. A necessary step in this direction is that of capturing and understanding group dynamics. In this paper, we discuss a particular scenario, in which meeting participants receive multimedia feedback on their relational behaviour, as a first step towards increasing self-awareness. We describe the background and the motivation for a coding scheme for annotating meeting recordings partially inspired by the Bales’ Interaction Process Analysis. This coding scheme was aimed at identifying suitable observable behavioural sequences. The study is complemented with an experimental investigation on the acceptability of such a service.
---
paper_title: Using linguistic cues for the automatic recognition of personality in conversation and text

paper_content:

It is well known that utterances convey a great deal of information about the speaker in addition to their semantic content. One such type of information consists of cues to the speaker's personality traits, the most fundamental dimension of variation between humans. Recent work explores the automatic detection of other types of pragmatic variation in text and conversation, such as emotion, deception, speaker charisma, dominance, point of view, subjectivity, opinion and sentiment. Personality affects these other aspects of linguistic production, and thus personality recognition may be useful for these tasks, in addition to many other potential applications. However, to date, there is little work on the automatic recognition of personality traits. This article reports experimental results for recognition of all Big Five personality traits, in both conversation and text, utilising both self and observer ratings of personality. While other work reports classification results, we experiment with classification, regression and ranking models. For each model, we analyse the effect of different feature sets on accuracy. Results show that for some traits, any type of statistical model performs significantly better than the baseline, but ranking models perform best overall. We also present an experiment suggesting that ranking models are more accurate than multi-class classifiers for modelling personality. In addition, recognition models trained on observed personality perform better than models trained using self-reports, and the optimal feature set depends on the personality trait. A qualitative analysis of the learned models confirms previous findings linking language and personality, while revealing many new linguistic markers.
---
paper_title: Interpersonal Adaptation: Dyadic Interaction Patterns

paper_content:

List of figures and tables Preface Part I. Overview: 1. Introduction Part II. Interaction Adaptation Theories and Models: 2. Biological approaches 3. Arousal and affect approaches 4. Social norm approaches 5. Communication and cognitive approaches Part III. Issues in Studying Interaction Adaptation: 6. Reconceptualising interaction adaptation patterns 7. Operationalising adaptation patterns 8. Analysing adaptation patterns Part IV. Multimethod Tests of Reciprocity and Compensation: 9. A first illustration 10. Further illustrations Part V. Developing a New Interpersonal Adaptation Theory: 11. The theories revisited 12. A research agenda References Index.
---
paper_title: Capturing Individual and Group Behavior with Wearable Sensors

paper_content:

We show how to obtain high level descriptions of human behavior in terms of physical activity, speech activity, face-to-face interaction (f2f), physical proximity, and social network attributes from sensor data. We present experimental results that showthat it is possible to identify individual personality traits as well as subjective and objective group performance metrics from low level data collected using wearable sensors.
---
paper_title: Detection and application of influence rankings in small group meetings

paper_content:

We address the problem of automatically detecting participant's influence levels in meetings. The impact and social psychological background are discussed. The more influential a participant is, the more he or she influences the outcome of a meeting. Experiments on 40 meetings show that application of statistical (both dynamic and static) models while using simply obtainable features results in a best prediction performance of 70.59% when using a static model, a balanced training set, and three discrete classes: high, normal and low. Application of the detected levels are shown in various ways i.e. in a virtual meeting environment as well as in a meeting browser system.
---
paper_title: The chameleon effect : The perception-behavior link and social interaction

paper_content:

The chameleon effect refers to nonconscious mimicry of the postures, mannerisms, facial expressions, and other behaviors of one's interaction partners, such that one's behavior passively and unintentionally changes to match that of others in one's current social environment. The authors suggest that the mechanism involved is Has. perception-behavior link, the recently documented finding (e.g., J. A. Bargh, M. Chen, & L. Burrows, 1996) that the mere perception of another's behavior automatically increases the likelihood of engaging in that behavior oneself. Experiment 1 showed that the motor behavior of participants unintentionally matched that of strangers with whom they worked on a task. Experiment 2 had confederates mimic the posture and movements of participants and showed that mimicry facilitates the smoothness of interactions and increases liking between interaction partners. Experiment 3 showed that dispositionally empathic individuals exhibit the chameleon effect to a greater extent than do other people.
---
paper_title: Spotting agreement and disagreement: A survey of nonverbal audiovisual cues and tools

paper_content:

While detecting and interpreting temporal patterns of non-verbal behavioral cues in a given context is a natural and often unconscious process for humans, it remains a rather difficult task for computer systems. Nevertheless, it is an important one to achieve if the goal is to realise a naturalistic communication between humans and machines. Machines that are able to sense social attitudes like agreement and disagreement and respond to them in a meaningful way are likely to be welcomed by users due to the more natural, efficient and human-centered interaction they are bound to experience. This paper surveys the nonverbal cues that could be present during agreement and disagreement behavioural displays and lists a number of tools that could be useful in detecting them, as well as a few publicly available databases that could be used to train these tools for analysis of spontaneous, audiovisual instances of agreement and disagreement.
---
paper_title: Thin Slices of Negotiation: Predicting outcomes from conversational dynamics within the first five minutes

paper_content:

In this research we examine whether conversational dynamics occurring within the first five minutes of a negotiation can predict negotiated outcomes. In a simulated employment negotiation, micro-coding conducted by a computer showed that activity level, conversational engagement, prosodic emphasis, and vocal mirroring predicted 30% of the variance in individual outcomes. The conversational dynamics associated with individual success among high-status parties were different from those associated with individual success among low-status parties. Results are interpreted in light of theory and research exploring the predictive power of "thin slices" (Ambady & Rosenthal, 1992). Implications include the development of new technology to diagnose and improve negotiation processes.
---
paper_title: A corpus for studying addressing behavior in multi-party dialogues

paper_content:

This paper describes a multi-modal corpus of hand-annotated meeting dialogues that was designed for studying addressing behavior in face-to-face conversations. The corpus contains annotated dialogue acts, addressees, adjacency pairs and gaze direction. First, we describe the corpus design where we present the annotation schema, annotation tools and annotation process itself. Then, we analyze the reproducibility and stability of the annotation schema.
---
paper_title: The behaviour markup language: recent developments and challenges

paper_content:

Since the beginning of the SAIBA effort to unify key interfaces in the multi-modal behavior generation process, the Behavior Markup Language (BML) has both gained ground as an important component in many projects worldwide, and continues to undergo further refinement. This paper reports on the progress made in the last year in further developing BML. It discusses some of the key challenges identified that the effort is facing, and reviews a number of projects that already are making use of BML or support its use.
---
paper_title: Automatic nonverbal analysis of social interaction in small groups : A review

paper_content:

An increasing awareness of the scientific and technological value of the automatic understanding of face-to-face social interaction has motivated in the past few years a surge of interest in the devising of computational techniques for conversational analysis. As an alternative to existing linguistic approaches for the automatic analysis of conversations, a relatively recent domain is using findings in social cognition, social psychology, and communication that have established the key role that nonverbal communication plays in the formation, maintenance, and evolution of a number of fundamental social constructs, which emerge from face-to-face interactions in time scales that range from short glimpses all the way to long-term encounters. Small group conversations are a specific case on which much of this work has been conducted. This paper reviews the existing literature on automatic analysis of small group conversations using nonverbal communication, and aims at bridging the current fragmentation of the work in this domain, currently split among half a dozen technical communities. The review is organized around the main themes studied in the literature and discusses, in a comparative fashion, about 100 works addressing problems related to the computational modeling of interaction management, internal states, personality traits, and social relationships in small group conversations, along with pointers to the relevant literature in social science. Some of the many open challenges and opportunities in this domain are also discussed.
---
paper_title: Canal9: A database of political debates for analysis of social interactions

paper_content:

Automatic analysis of social interactions attracts major attention in the computing community, but relatively few benchmarks are available to researchers active in the domain. This paper presents a new, publicly available, corpus of political debates including not only raw data, but a rich set of socially relevant annotations such as turn-taking (who speaks when and how much), agreement and disagreement between participants, and role played by people involved in each debate. The collection includes 70 debates for a total of 43 hours and 10 minutes of material.
---
paper_title: The SEMAINE API: Towards a standards-based framework for building emotion-oriented systems

paper_content:

This paper presents the SEMAINE API, an open source framework for building emotion-oriented systems. By encouraging and simplifying the use of standard representation formats, the framework aims to contribute to interoperability and reuse of system components in the research community. By providing a Java and C++ wrapper around a message-oriented middleware, the API makes it easy to integrate components running on different operating systems and written in different programming languages. The SEMAINE system 1.0 is presented as an example of a full-scale system built on top of the SEMAINE API. Three small example systems are described in detail to illustrate how integration between existing and new components is realised with minimal effort.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: Detection and application of influence rankings in small group meetings

paper_content:

We address the problem of automatically detecting participant's influence levels in meetings. The impact and social psychological background are discussed. The more influential a participant is, the more he or she influences the outcome of a meeting. Experiments on 40 meetings show that application of statistical (both dynamic and static) models while using simply obtainable features results in a best prediction performance of 70.59% when using a static model, a balanced training set, and three discrete classes: high, normal and low. Application of the detected levels are shown in various ways i.e. in a virtual meeting environment as well as in a meeting browser system.
---
paper_title: The AMI Meeting Corpus: A Pre-announcement

paper_content:

The AMI Meeting Corpus is a multi-modal data set consisting of 100 hours of meeting recordings. It is being created in the context of a project that is developing meeting browsing technology and will eventually be released publicly. Some of the meetings it contains are naturally occurring, and some are elicited, particularly using a scenario in which the participants play different roles in a design team, taking a design project from kick-off to completion over the course of a day. The corpus is being recorded using a wide range of devices including close-talking and far-field microphones, individual and room-view video cameras, projection, a whiteboard, and individual pens, all of which produce output signals that are synchronized with each other. It is also being hand-annotated for many different phenomena, including orthographic transcription, discourse properties such as named entities and dialogue acts, summaries, emotions, and some head and hand gestures. We describe the data set, including the rationale behind using elicited material, and explain how the material is being recorded, transcribed and annotated.
---
paper_title: Automatic analysis of multimodal group actions in meetings

paper_content:

This paper investigates the recognition of group actions in meetings. A framework is employed in which group actions result from the interactions of the individual participants. The group actions are modeled using different HMM-based approaches, where the observations are provided by a set of audiovisual features monitoring the actions of individuals. Experiments demonstrate the importance of taking interactions into account in modeling the group actions. It is also shown that the visual modality contains useful information, even for predominantly audio-based events, motivating a multimodal approach to meeting analysis.
---
paper_title: Multimedia Database of Meetings and Informal Interactions for Tracking Participant Involvement and Discourse Flow

paper_content:

At ATR, we are collecting and analysing ‘meetings’ data using a table-top sensor device consisting of a small 360-degree camera surrounded by an array of high-quality directional microphones. This equipment provides a stream of information about the audio and visual events of the meeting which is then processed to form a representation of the verbal and non-verbal interpersonal activity, or discourse flow, during the meeting. This paper describes the resulting corpus of speech and video data which is being collected for the above research. It currently includes data from 12 monthly sessions, comprising 71 video and 33 audio modules. Collection is continuing monthly and is scheduled to include another ten sessions.
---
paper_title: VACE Multimodal Meeting Corpus

paper_content:

In this paper, we report on the infrastructure we have developed to support our research on multimodal cues for understanding meetings. With our focus on multimodality, we investigate the interaction among speech, gesture, posture, and gaze in meetings. For this purpose, a high quality multimodal corpus is being produced.
---
paper_title: Human computing and machine understanding of human behavior: a survey

paper_content:

A widely accepted prediction is that computing will move to the background, weaving itself into the fabric of our everyday living spaces and projecting the human user into the foreground. If this prediction is to come true, then next generation computing, which we will call human computing, should be about anticipatory user interfaces that should be human-centered, built for humans based on human models. They should transcend the traditional keyboard and mouse to include natural, human-like interactive functions including understanding and emulating certain human behaviors such as affective and social signaling. This article discusses a number of components of human behavior, how they might be integrated into computers, and how far we are from realizing the front end of human computing, that is, how far are we from enabling computers to understand human behavior.
---
paper_title: Multimodal databases of everyday emotion: facing up to complexity.

paper_content:

In everyday life, speech is part of a multichannel system involved in conveying emotion. Understanding how it operates in that context requires suitable data, consisting of multimodal records of emotion drawn from everyday life. This paper reflects the experience of two teams active in collecting and labelling data of this type. It sets out the core reasons for pursuing a multimodal approach, reviews issues and problems for developing relevant databases, and indicates how we can move forward both in terms of data collection and approaches to labelling.
---
paper_title: Culture and social behavior

paper_content:

Our culture influences who we are and our understanding of social behaviour why bother to study culture-social behaviour relationships how to study cultures the analysis of subjective cultures some interesting differences in subjective cultures cultural differences in patterns of social behaviour culture and communication cultural differences in aggression, helping, dominance, conformity, obedience and intimacy dealing with diversity and intercultural relations intercultural training.
---
paper_title: A survey of affect recognition methods: audio, visual and spontaneous expressions

paper_content:

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. Promising approaches have been reported, including automatic methods for facial and vocal affect recognition. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions-despite the fact that deliberate behavior differs in visual and audio expressions from spontaneously occurring behavior. Recently efforts to develop algorithms that can process naturally occurring human affective behavior have emerged. This paper surveys these efforts. We first discuss human emotion perception from a psychological perspective. Next, we examine the available approaches to solving the problem of machine understanding of human affective behavior occurring in real-world settings. We finally outline some scientific and engineering challenges for advancing human affect sensing technology.
---
paper_title: A new image for fear and emotion

paper_content:

We experience many emotions every day, but as a topic of study emotion has been neglected until recently. Now, however, with the development of sophisticated techniques such as functional magnetic resonance imaging, the neural basis of emotion can be studied. Four new reports have used slightly different approaches, and conclude that the amygdala — a complex structure within the temporal lobes of the brain — is involved in the responses to fear.
---
paper_title: Speech-driven lip motion generation with a trajectory HMM

paper_content:

Automatic speech animation remains a challenging problem that can be described as finding the optimal sequence of animation parameter configurations given some speech. In this paper we present a novel technique to automatically synthesise lip motion trajectories from a speech signal. The developed system predicts lip motion units from the speech signal and generates animation trajectories automatically employing a ”Trajectory Hidden Markov Model”. Using the MLE criterion, its parameter generation algorithm produces the optimal smooth motion trajectories that are used to drive control points on the lips directly. Additionally, experiments were carried out to find a suitable model unit that produces the most accurate results. Finally a perceptual evaluation was conducted, that showed that the developed motion units perform better than phonemes.
---
paper_title: Nonverbal Communication in Human Interaction

paper_content:

Preface. Part I: AN INTRODUCTION TO THE STUDY OF NONVERBAL COMMUNICATION. 1. Nonverbal Communication: Basic Perspectives. 2. The Roots of Nonverbal Behavior. 3. The Ability to Receive and Send Nonverbal Signals. Part II: THE COMMUNICATION ENVIRONMENT. 4. The Effects of the Environment on Human Communication. 5. The Effects of Territory and Personal Space on Human Communication. Part III: THE COMMUNICATORS. 6. The Effects of Physical Characteristics on Human Communication. Part IV: The Communicators' Behavior. 7. The Effects of Gesture and Posture on Human Communication. 8. The Effects of Touch on Human Communication. 9. The Effects of the Face on Human Communication. 10. The Effects of Eye Behavior on Human Communication. 11. The Effects of Vocal Cues That Accompany Spoken Words. Part V: COMMUNICATING IMPORTANT MESSAGES. 12. Using Nonverbal Behavior in Daily Interaction. 13. Nonverbal Messages in Special Contexts.
---
paper_title: The Significance of Posture in Communication Systems

paper_content:

Devices relying on the Inverse Wiedman Effect. Devices include a current conductive, magnetically anisotropic rod, through which an AC current flows. Wound about the rod is a conductive coil having output terminals. By varying the AC current, the anisotropy, or both, variations in the output across coil terminals can be obtained. Devices may be for current sensing, pressure sensing, fluid flow sensing or an electric push button.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: A survey of affect recognition methods: audio, visual and spontaneous expressions

paper_content:

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. Promising approaches have been reported, including automatic methods for facial and vocal affect recognition. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions-despite the fact that deliberate behavior differs in visual and audio expressions from spontaneously occurring behavior. Recently efforts to develop algorithms that can process naturally occurring human affective behavior have emerged. This paper surveys these efforts. We first discuss human emotion perception from a psychological perspective. Next, we examine the available approaches to solving the problem of machine understanding of human affective behavior occurring in real-world settings. We finally outline some scientific and engineering challenges for advancing human affect sensing technology.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: Facial and Vocal Expressions of Emotion

paper_content:

A flurry of theoretical and empirical work concerning the production of and response to facial and vocal expressions has occurred in the past decade. That emotional expressions express emotions is a tautology but may not be a fact. Debates have centered on universality, the nature of emotion, and the link between emotions and expressions. Modern evolutionary theory is informing more models, emphasizing that expressions are directed at a receiver, that the interests of sender and receiver can conflict, that there are many determinants of sending an expression in addition to emotion, that expressions influence the receiver in a variety of ways, and that the receiver's response is more than simply decoding a message.
---
paper_title: A survey of affect recognition methods: audio, visual and spontaneous expressions

paper_content:

Automated analysis of human affective behavior has attracted increasing attention from researchers in psychology, computer science, linguistics, neuroscience, and related disciplines. Promising approaches have been reported, including automatic methods for facial and vocal affect recognition. However, the existing methods typically handle only deliberately displayed and exaggerated expressions of prototypical emotions-despite the fact that deliberate behavior differs in visual and audio expressions from spontaneously occurring behavior. Recently efforts to develop algorithms that can process naturally occurring human affective behavior have emerged. This paper surveys these efforts. We first discuss human emotion perception from a psychological perspective. Next, we examine the available approaches to solving the problem of machine understanding of human affective behavior occurring in real-world settings. We finally outline some scientific and engineering challenges for advancing human affect sensing technology.
---
paper_title: Embodiment in conversational interfaces: Rea

paper_content:

In this paper, we argue for embodied corrversational charactersas the logical extension of the metaphor of human - computerinteraction as a conversation. We argue that the only way to fullymodel the richness of human I&+ to-face communication is torely on conversational analysis that describes sets ofconversational behaviors as fi~lfilling conversational functions,both interactional and propositional. We demonstrate how toimplement this approach in Rea, an embodied conversational agentthat is capable of both multimodal input understanding and outputgeneration in a limited application domain. Rea supports bothsocial and task-oriented dialogue. We discuss issues that need tobe addressed in creating embodied conversational agents, anddescribe the architecture of the Rea interface.
---
paper_title: How (Not) to Add Laughter to Synthetic Speech

paper_content:

Laughter is a powerful means of emotion expression which has not yet been used in speech synthesis. The current paper reports on a pilot study in which differently created types of laughter were combined with synthetic speech in a dialogical situation. A perception test assessed the effect on perceived social bonding as well as the appropriateness of the laughter. Results indicate that it is crucial to carefully model the intensity of the laughter, whereas speaker identity and generation method appear less important.
---
paper_title: Social force model for pedestrian dynamics

paper_content:

It is suggested that the motion of pedestrians can be described as if they would be subject to ``social forces.'' These ``forces'' are not directly exerted by the pedestrians' personal environment, but they are a measure for the internal motivations of the individuals to perform certain actions (movements). The corresponding force concept is discussed in more detail and can also be applied to the description of other behaviors. In the presented model of pedestrian behavior several force terms are essential: first, a term describing the acceleration towards the desired velocity of motion; second, terms reflecting that a pedestrian keeps a certain distance from other pedestrians and borders; and third, a term modeling attractive effects. The resulting equations of motion of nonlinearly coupled Langevin equations. Computer simulations of crowds of interacting pedestrians show that the social force model is capable of describing the self-organization of several observed collective effects of pedestrian behavior very realistically.
---
paper_title: Towards a virtual agent using similarity-based laughter production

paper_content:

Facial expressions of emotions are often described statically at their apex. Lately several researchers (Keltner, 1995) have shown through analysis of video corpora that emotions are expressed through a sequence of micro-behaviours. These micro-behaviours correspond to signals spread over the whole body (face, head, gaze, gesture, etc). All these signals do not have to occur simultaneously. Some of them occur more often at the beginning of expressions, some others at the end, some occur in a sequence, while others are synchronized (Keltner, 1995).
---
paper_title: D.: Incremental multimodal feedback for conversational agents

paper_content:

Just like humans, conversational computer systems should not listen silently to their input and then respond. Instead, they should enforce the speaker-listener link by attending actively and giving feedback on an utterance while perceiving it. Most existing systems produce direct feedback responses to decisive (e.g. prosodic) cues. We present a framework that conceives of feedback as a more complex system, resulting from the interplay of conventionalized responses to eliciting speaker events and the multimodal behavior that signals how internal states of the listener evolve. A model for producing such incremental feedback, based on multi-layered processes for perceiving, understanding, and evaluating input, is described.
---
paper_title: Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents

paper_content:

We describe an implemented system which automatically generates and animates conversations between multiple human-like agents with appropriate and synchronized speech, intonation, facial expressions, and hand gestures. Conversation is created by a dialogue planner that produces the text as well as the intonation of the utterances. The speaker/listener relationship, the text, and the intonation in turn drive facial expressions, lip motions, eye gaze, head motion, and arm gestures generators. Coordinated arm, wrist, and hand motions are invoked to create semantically meaningful gestures. Throughout we will use examples from an actual synthesized, fully animated conversation.
---
paper_title: Automatic acoustic synthesis of human-like laughter.

paper_content:

A technique to synthesize laughter based on time-domain behavior of real instances of human laughter is presented. In the speech synthesis community, interest in improving the expressive quality of synthetic speech has grown considerably. While the focus has been on the linguistic aspects, such as precise control of speech intonation to achieve desired expressiveness, inclusion of nonlinguistic cues could further enhance the expressive quality of synthetic speech. Laughter is one such cue used for communicating, say, a happy or amusing context. It can be generated in many varieties and qualities: from a short exhalation to a long full-blown episode. Laughter is modeled at two levels, the overall episode level and at the local call level. The first attempts to capture the overall temporal behavior in a parametric model based on the equations that govern the simple harmonic motion of a mass-spring system is presented. By changing a set of easily available parameters, the authors are able to synthesize a var...
---
paper_title: Analyzing single episodes of interaction: an exercise in conversation analysis

paper_content:

A variety of analytic resources provided by past work in conversation analysis are brought to bear on the analysis of a single utterance in its sequential context, drawn from an ordinary conversation. Various facets of the organization of talk-in-interaction are thereby both introduced and exemplified. The result displays the capacity of this analytic modality to meet a fundamental responsibility of social analysis, namely the capacity to explicate single episodes of action in interaction as a basic locus of social order.
---
paper_title: Annotating meaning of listener vocalizations for speech synthesis

paper_content:

Generation of listener vocalizations is one of the major objectives of emotionally colored conversational speech synthesis. Success in this endeavor depends on the answers to three questions: What kinds of meaning are expressed through listener vocalizations? What form is suitable for a given meaning? And, in what context should which listener vocalizations be produced? In this paper, we address the first of these questions. We present a method to record natural and expressive listener vocalizations for synthesis, and describe our approach to identify a suitable categorical description of the meaning conveyed in the vocalizations. In our data, one actor produces a total of 967 listener vocalizations, in his natural speaking style and three acted emotion-specific personalities. In an open categorization scheme, we find that eleven categories occur on at least 5% of the vocalizations, and that most vocalizations are better described by two or three categories rather than a single one. Furthermore, an annotation of meaning reference, according to Bu¨hler's Organon model, allows us to make interesting observations regarding the listener's own state, his stance towards the interlocutor, and his attitude towards the topic of the conversation.
---
paper_title: Human–machine interaction as a model of machine–machine interaction: how to make machines interact as humans do

paper_content:

Turn-taking is one of the main features of communicative systems. In particular, it is one of the bases allowing robust interactions in imitation, thanks to its two linked aspects, i.e., communication and learning. In this article, we propose a simple model based on the interaction of two neural oscillators inhibiting each other which explain how 'turn-taking' may emerge dynamically between two agents. An implementation of the model on a simple robotic platform made of one CCD camera and one simple arm (ADRIANA platform) is detailed. Results showing the emergence of a 'turn-taking' dynamics on this platform are discussed and an extension in simulation for a larger scale of parameters in order to validate robustness is given.
---
paper_title: Dynamic Movement and Positioning of Embodied Agents in Multiparty Conversations

paper_content:

For embodied agents to engage in realistic multiparty conversation, they must stand in appropriate places with respect to other agents and the environment. When these factors change, for example when an agent joins a conversation, the agents must dynamically move to a new location and/or orientation to accommodate. This paper presents an algorithm for simulating the movement of agents based on observed human behavior using techniques developed for pedestrian movement in crowd simulations. We extend a previous group conversation simulation to include an agent motion algorithm. We examine several test cases and show how the simulation generates results that mirror real-life conversation settings.
---
paper_title: Perception of non-verbal emotional listener feedback

paper_content:

This paper reports on a listening test assessing the perception of short non-verbal emotional vocalisations emitted by a listener as feedback to the speaker. We clarify the concepts backchannel and feedback, and investigate the use of affect bursts as a means of giving emotional feedback via the backchannel. Experiments with German and Dutch subjects confirm that the recognition of emotion from affect bursts in a dialogical context is similar to their perception in isolation. We also investigate the acceptability of affect bursts when used as listener feedback. Acceptability appears to be linked to display rules for emotion expression. While many ratings were similar between Dutch and German listeners, a number of clear differences was found, suggesting language-specific affect bursts.
---
paper_title: Natural Behavior of a Listening Agent

paper_content:

In contrast to the variety of listening behaviors produced in human-to-human interaction, most virtual agents sit or stand passively when a user speaks. This is a reflection of the fact that although the correct responsive behavior of a listener during a conversation is often related to the semantics, the state of current speech understanding technology is such that semantic information is unavailable until after an utterance is complete. This paper will illustrate that appropriate listening behavior can also be generated by other features of a speaker's behavior that are available in real time such as speech quality, posture shifts and head movements. This paper presents a mapping from these real-time obtainable features of a human speaker to agent listening behaviors.
---
paper_title: Predicting listener backchannels: A probabilistic multimodal approach

paper_content:

During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model or Conditional Random Fields) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.
---
paper_title: Expressing degree of activation in synthetic speech

paper_content:

This paper presents the design, implementation, and evaluation of a system capable of expressing a continuum of emotional states in synthetic speech. A review of the literature and an analysis of a naturalistic database of emotional speech provided detailed descriptions of the link between acoustic parameters and the three emotion dimensions activation, evaluation, and power. We formulated a set of emotional prosody rules and implemented them in a German text-to-speech (TTS) system. A perception study investigated how well the resulting synthesized prosody fits with emotional states defined through textual situation descriptions. Results show that degree of activation is perceived as intended
---
paper_title: Modelling personality features by changing prosody in synthetic speech

paper_content:

This study explores how features of brand personalities can b e modelled with the prosodic parameters pitch level, pitch range, articulation rate and loudness. Experiments with parametric al diphone synthesis showed that listeners rated the prosodically changed versions better than a baseline version for the dimen- sions "sincerity", "competence", "sophistication", "excitem ent" and "ruggedness". The contribution of prosodic features such as lower pitch and an enlarged pitch range are analyzed and discussed.
---
paper_title: Verification of Acoustical Correlates of Emotional Speech using Formant- Synthesis

paper_content:

This paper explores the perceptual relevance of acoustical correlates of emotional speech by means of speech synthesis. Besides, the research aims at the development of emotion-rules which enable an optimized speech synthesis system to generate emotional speech. Two invesigations using this synthetizer are described : 1) the systematic variation of selected acoustical features to gain a preliminary impression regarding the importance of certain acoustical features for emotional expression, and 2) the specific manipulation of a stimulus spoken under emotionally neutral condition to investigate further the effect of certain features and the overall ability of the synthetizer to generate recognizable emotional expression. It is shown that this approach is indeed capable of generating emotional speech that is recognized almost as well as utterances realized by actors
---
paper_title: Expressive Speech Synthesis: Past, Present, and Possible Futures

paper_content:

Approaches towards adding expressivity to synthetic speech have changed considerably over the last 20 years. Early systems, including formant and diphone systems, have been focused around “explicit control” models; early unit selection systems have adopted a “playback” approach. Currently, various approaches are being pursued to increase the flexibility in expression while maintaining the quality of state-of-the-art systems, among them a new “implicit control” paradigm in statistical parametric speech synthesis, which provides control over expressivity by combining and interpolating between statistical models trained on different expressive databases. The present chapter provides an overview of the past and present approaches, and ventures a look into possible future developments.
---
paper_title: Automatic Exploration of Corpus-Specific Properties for Expressive Text-to-Speech: A Case Study in Emphasis.

paper_content:

In this paper we explore an approach to expressive text-tospeech synthesis in which pre-existing expression-specific corpora are complemented with automatically generated labels to augment the search space of units the engine can exploit to increase its expressiveness. We motivate this data-discovery approach as an alternative to an approach guided by data collection, in order to harness the full usefulness of the expressiveness already contained in a synthesis corpus. We illustrate the approach with a case study that uses emphasis as its intended expression, describe algorithms for the automatic discovery of such instances in the database and how to make use of them during synthesis, and, finally, evaluate the benefits of the proposal to demonstrate the feasibility of the approach.
---
paper_title: Endowing spoken language dialogue systems with emotional intelligence

paper_content:

While most dialogue systems restrict themselves to the adjustment of the propositional contents, our work concentrates on the generation of stylistic va- riations in order to improve the user's perception of the interaction. To accomplish this goal, our approach integrates a social theory of politeness with a cognitive theory of emotions. We propose a hierarchical selection process for politeness behaviors in order to enable the refinement of decisions in case additional context information becomes available.
---
paper_title: Model Adaptation Approach to Speech Synthesis with Diverse Voices and Styles

paper_content:

In human computer interaction and dialogue systems, it is often desirable for text-to-speech synthesis to be able to generate natural sounding speech with an arbitrary speaker's voice and with varying speaking styles and/or emotional expressions. We have developed an average-voice-based speech synthesis method using statistical average voice models and model adaptation techniques for this purpose. In this paper, we describe an overview of the speech synthesis system and show the current performance with several experimental results.
---
paper_title: Simultaneous Modeling Of Spectrum, Pitch And Duration In HMM-Based Speech Synthesis

paper_content:

In this paper, we describe an HMM-based speech synthesis system in which spectrum, pitch and state duration are modeled simultaneously in a unified framework of HMM. In the system, pitch and state duration are modeled by multi-space probability distribution HMMs and multi-dimensional Gaussian distributions, respectively. The distributions for spectral parameter, pitch parameter and the state duration are clustered independently by using a decision-tree based context clustering technique. Synthetic speech is generated by using an speech parameter generation algorithm from HMM and a mel-cepstrum based vocoding technique. Through informal listening tests, we have confirmed that the proposed system successfully synthesizes natural-sounding speech which resembles the speaker in the training database.
---
paper_title: The IBM expressive text-to-speech synthesis system for American English

paper_content:

Expressive text-to-speech (TTS) synthesis should contribute to the pleasantness, intelligibility, and speed of speech-based human-machine interactions which use TTS. We describe a TTS engine which can be directed, via text markup, to use a variety of expressive styles, here, questioning, contrastive emphasis, and conveying good and bad news. Differences in these styles lead us to investigate two approaches for expressive TTS, a "corpus-driven" and a "prosodic-phonology" approach. Each speaker records 11 h (excluding silences) of "neutral" sentences. In the corpus-driven approach, the speaker also records 1-h corpora in each expressive style; these segments are tagged by style for use during search, and decision trees for determining f0 contours and timing are trained separately for each of the neutral and expressive corpora. In the prosodic-phonology approach, rules translating certain expressive markup elements to tones and break indices (ToBI) are manually determined, and the ToBI elements are used in single f0 and duration trees for all expressions. Tests show that listeners identify synthesis in particular styles ranging from 70% correctly for "conveying bad news" to 85% for "yes-no questions". Further improvements are demonstrated through the use of speaker-pooled f0 and duration models
---
paper_title: Politeness : some universals in language usage

paper_content:

Symbols and abbreviations Foreword John J. Gumperz Introduction to the reissue Notes 1. Introduction 2. Summarized argument 3. The argument: intuitive bases and derivative definitions 4. On the nature of the model 5. Realizations of politeness strategies in language 6. Derivative hypotheses 7. Sociological implications 8. Implications for language studies 9. Conclusions Notes References Author index Subject index.
---
paper_title: Modelling politeness in natural language generation

paper_content:

One of the main objectives of research in Natural Language generation (NLG) is to account for linguistic variation in a systematic way. Research on linguistic politeness provides important clues as to the possible causes of linguistic variation and the ways in which it may be modelled formally. In this paper we present a simple language generation model for choosing the appropriate surface realisations of tutoring responses based on the politeness notion of face. We adapt the existing definition of face to the demands of the educational genre and we demonstrate how a politeness driven NLG system may result in a more natural and a more varied form of linguistic output.
---
paper_title: Modelling prominence and emphasis improves unit-selection synthesis

paper_content:

We describe the results of large scale perception experiments showing improvements in synthesising two distinct kinds of prominence: standard pitch-accent and strong emphatic accents. Previously prominence assignment has been mainly evaluated by computing accuracy on a prominence-labelled test set. By contrast we integrated an automatic pitch-accent classifie r into the unit selection target cost and showed that listeners preferred these synthesised sentences. We also describe an improved recording script for collecting emphatic accents, and show that generating emphatic accents leads to further improvements in the fict ion genre over incorporating pitch accent only. Finally, we show diffe rences in the effe cts of prominence between childdirected speech and news and fict ion genres.
---
paper_title: A style control technique for HMM-based speech synthesis.

paper_content:

This paper describes an approach to controlling style of synthetic speech in HMM-based speech synthesis. The style is defined as one of speaking styles and emotional expressions in speech. We model each speech synthesis unit by using a context-dependent HMM whose mean vector of the output distribution function is given by a function of a parameter vector called style control vector. We assume that the mean vector is modeled by multiple regression with the style control vector. The multiple regression matrices are estimated by EMalgorithm as well as other model parameters of HMMs. In the synthesis stage, the mean vectors are modified by transforming an arbitrarily given control vector which is associated with a desired style. The results of subjective tests show that we can control styles by choosing the style control vector appropriately.
---
paper_title: Generating Socially Appropriate Tutorial Dialog

paper_content:

Analysis of student-tutor coaching dialogs suggest that good human tutors attend to and attempt to influence the motivational state of learners. Moreover, they are sensitive to the social face of the learner, and seek to mitigate the potential face threat of their comments. This paper describes a dialog generator for pedagogical agents that takes motivation and face threat factors into account. This enables the agent to interact with learners in a socially appropriate fashion, and foster intrinsic motivation on the part of the learner, which in turn may lead to more positive learner affective states.
---
paper_title: Politeness and Alignment in Dialogues with a Virtual Guide

paper_content:

Language alignment is something that happens automatically in dialogues between human speakers. The ability to align is expected to increase the believability of virtual dialogue agents. In this paper we extend the notion of alignment to affective language use, describing a model for dynamically adapting the linguistic style of a virtual agent to the level of politeness and formality detected in the user's utterances. The model has been implemented in the Virtual Guide, an embodied conversational agent giving directions in a virtual environment. Evaluation shows that our formality model needs improvement, but that the politeness tactics used by the Guide are mostly interpreted as intended, and that the alignment to the user's language is noticeable.
---
paper_title: Unit selection in a concatenative speech synthesis system using a large speech database

paper_content:

One approach to the generation of natural-sounding synthesized speech waveforms is to select and concatenate units from a large speech database. Units (in the current work, phonemes) are selected to produce a natural realisation of a target phoneme sequence predicted from text which is annotated with prosodic and phonetic context information. We propose that the units in a synthesis database can be considered as a state transition network in which the state occupancy cost is the distance between a database unit and a target, and the transition cost is an estimate of the quality of concatenation of two consecutive units. This framework has many similarities to HMM-based speech recognition. A pruned Viterbi search is used to select the best units for synthesis from the database. This approach to waveform synthesis permits training from natural speech: two methods for training from speech are presented which provide weights which produce more natural speech than can be obtained by hand-tuning.
---
paper_title: Social role awareness in animated agents

paper_content:

This paper promotes {\itshape social role awareness\/} as a desirable capability of animated agents, that are by now strong affective reasoners, but otherwise often lack the social competence observed with humans. In particular, humans may easily adjust their behavior depending on their respective role in a socio-organizational setting, whereas their synthetic pendants tend to be driven mostly by attitudes, emotions, and personality. Our main contribution is the incorporation of `social filter programs' to mental models of animated agents. Those programs may qualify an agent's expression of its emotional state by the social context, thereby enhancing the agent's believability as a conversational partner or virtual teammate. Our implemented system is entirely web-based and demonstrates socially aware animated agents in an environment similar to Hayes-Roth's Cybercaf\'{e}.
---
paper_title: 2007b Model of facial expressions management for an embodied conversational agent

paper_content:

In this paper we present a model of facial behaviour encompassing interpersonal relations for an Embodied Conversational Agent (ECA). Although previous solutions of this problem exist in ECA's domain, in our approach a variety of facial expressions (i.e. expressed, masked, inhibited, and fake expressions) is used for the first time. Moreover, our rules of facial behaviour management are consistent with the predictions of politeness theory as well as the experimental data (i.e. annotation of the video-corpus). Knowing the affective state of the agent and the type of relations between interlocutors the system automatically adapts the facial behaviour of an agent to the social context. We present also the evaluation study we have conducted of our model. In this experiment we analysed the perception of interpersonal relations from the facial behaviour of our agent.
---
paper_title: Limited domain synthesis of expressive military speech for animated characters

paper_content:

Text-to-speech synthesis can play an important role in interactive education and training applications, as voices for animated agents. Such agents need high-quality voices capable of expressing intent and emotion. This paper presents preliminary results in an effort aimed at synthesizing expressive military speech for training applications. Such speech has acoustic and prosodic characteristics that can differ markedly from ordinary conversational speech. A limited domain synthesis approach is used employing samples of expressive speech, classified according to speaking style. The resulting synthesizer was tested both in isolation and in the context of a virtual reality training scenario with animated characters.
---
paper_title: Generating Politeness in Task Based Interaction: an Evaluation of the Effect of Linguistic Form and Culture

paper_content:

Politeness is an integral part of human language variation, e.g. consider the difference in the pragmatic effect of realizing the same communicative goal with either "Get me a glass of water mate!" or "I wonder if I could possibly have some water please?" This paper presents POLLy (Politeness for Language Learning), a system which combines a natural language generator with an AI Planner to model Brown and Levinson's theory of politeness (BL (2) our indirect strategies which should be the politest forms, are seen as the rudest; and (3) English and Indian native speakers of English have different perceptions of the level of politeness needed to mitigate particular face threats.
---
paper_title: IDEAS4Games: Building Expressive Virtual Characters for Computer Games

paper_content:

In this paper we present two virtual characters in an interactive poker game using RFID-tagged poker cards for the interaction. To support the game creation process, we have combined models, methods, and technology that are currently investigated in the ECA research field in a unique way. A powerful and easy-to-use multimodal dialog authoring tool is used for the modeling of game content and interaction. The poker characters rely on a sophisticated model of affect and a state-of-the art speech synthesizer. During the game, the characters show a consistent expressive behavior that reflects the individually simulated affect in speech and animations. As a result, users are provided with an engaging interactive poker experience.
---
paper_title: People as Flexible Interpreters: Evidence and Issues from Spontaneous Trait Inference

paper_content:

Publisher Summary This chapter investigates the ways in which readily inferences about others occur when inferences are not the focal task. The evidence and issues from spontaneous trait inference (STI) are also discussed in the chapter. STI occurs when attending to another person's behavior produces a trait inference in the absence of explicit intention to infer traits or form an impression of that person. Seven different paradigms have been employed to detect and investigate spontaneous trait inference: (1) cued recall under memory instructions; (2) cued recall of distractors; (3) recognition probe; (4) lexical decision; (5) delayed recognition; (6) word stem completion; and (7) relearning. Informational conditions review the ways in which the trait-relevant information presented in STI studies is systematically varied and its effects. The treatment of cognitive conditions focuses on the efficiency of STI and its minimal demands on cognitive capacity. The motivational conditions are divided into: proximal and distal goals. The chapter also explores whether STI refers to actors or merely to behaviors and the consequences of STI based on awareness, priming, prediction, and correspondence bias.
---
paper_title: Spontaneous Inferences, Implicit Impressions, and Implicit Theories

paper_content:

People make social inferences without intentions, awareness, or effort, i.e., spontaneously. We review recent findings on spontaneous social inferences (especially traits, goals, and causes) and closely related phenomena. We then describe current thinking on some of the most relevant processes, implicit knowledge, and theories. These include automatic and controlled processes and their interplay; embodied cognition, including mimicry; and associative versus rule-based processes. Implicit knowledge includes adult folk theories, conditions of personhood, self-knowledge to simulate others, and cultural and social class differences. Implicit theories concern Bayesian networks, recent attribution research, and questions about the utility of the disposition-situation dichotomy. Developmental research provides new insights. Spontaneous social inferences include a growing array of phenomena, but they have been insufficiently linked to other phenomena and theories. We hope the links suggested in this review begin to remedy this.
---
paper_title: Social Signal Processing: Survey of an Emerging Domain

paper_content:

The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life. This paper argues that next-generation computing needs to include the essence of social intelligence - the ability to recognize human social signals and social behaviours like turn taking, politeness, and disagreement - in order to become more effective and more efficient. Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, and similar, design and development of automated systems for social signal processing (SSP) are rather difficult. This paper surveys the past efforts in solving these problems by a computer, it summarizes the relevant findings in social psychology, and it proposes a set of recommendations for enabling the development of the next generation of socially aware computing.
---
paper_title: Expressive Speech Synthesis: Past, Present, and Possible Futures

paper_content:

Approaches towards adding expressivity to synthetic speech have changed considerably over the last 20 years. Early systems, including formant and diphone systems, have been focused around “explicit control” models; early unit selection systems have adopted a “playback” approach. Currently, various approaches are being pursued to increase the flexibility in expression while maintaining the quality of state-of-the-art systems, among them a new “implicit control” paradigm in statistical parametric speech synthesis, which provides control over expressivity by combining and interpolating between statistical models trained on different expressive databases. The present chapter provides an overview of the past and present approaches, and ventures a look into possible future developments.
---
paper_title: The behaviour markup language: recent developments and challenges

paper_content:

Since the beginning of the SAIBA effort to unify key interfaces in the multi-modal behavior generation process, the Behavior Markup Language (BML) has both gained ground as an important component in many projects worldwide, and continues to undergo further refinement. This paper reports on the progress made in the last year in further developing BML. It discusses some of the key challenges identified that the effort is facing, and reviews a number of projects that already are making use of BML or support its use.
---
paper_title: Comparing Rule-Based and Data-Driven Selection of Facial Displays

paper_content:

The non-verbal behaviour of an embodied conversational agent is normally based on recorded human behaviour. There are two main ways that the mapping from human behaviour to agent behaviour has been implemented. In some systems, human behaviour is analysed, and then rules for the agent are created based on the results of that analysis; in others, the recorded behaviour is used directly as a resource for decision-making, using data-driven techniques. In this paper, we implement both of these methods for selecting the conversational facial displays of an animated talking head and compare them in two user evaluations. In the first study, participants were asked for subjective preferences: they tended to prefer the output of the data-driven strategy, but this trend was not statistically significant. In the second study, the data-driven facial displays affected the ability of users to perceive user-model tailoring in synthesised speech, while the rule-based displays did not have any effect.
---
paper_title: The perception of emotions by ear and by eye

paper_content:

Emotions are expressed in the voice as well as on the face. As a first step to explore the question of their integration, we used a bimodal perception situation modelled after the McGurk paradigm, in which varying degrees of discordance can be created between the affects expressed in a face and in a tone of voice. Experiment 1 showed that subjects can effectively combine information from the two sources, in that identification of the emotion in the face is biased in the direction of the simultaneously presented tone of voice. Experiment 2 showed that this effect occurs also under instructions to base the judgement exclusively on the face. Experiment 3 showed the reverse effect, a bias from the emotion in the face on judgement of the emotion in the voice. These results strongly suggest the existence of mandatory bidirectional links between affect detection structures in vision and audition.
---
paper_title: Honest Signals: How They Shape Our World

paper_content:

How can you know when someone is bluffing? Paying attention? Genuinely interested? The answer, writes Sandy Pentland in Honest Signals, is that subtle patterns in how we interact with other people reveal our attitudes toward them. These unconscious social signals are not just a back channel or a complement to our conscious language; they form a separate communication network. Biologically based "honest signaling," evolved from ancient primate signaling mechanisms, offers an unmatched window into our intentions, goals, and values. If we understand this ancient channel of communication, Pentland claims, we can accurately predict the outcomes of situations ranging from job interviews to first dates. Pentland, an MIT professor, has used a specially designed digital sensor worn like an ID badgea "sociometer"to monitor and analyze the back-and-forth patterns of signaling among groups of people. He and his researchers found that this second channel of communication, revolving not around words but around social relations, profoundly influences major decisions in our liveseven though we are largely unaware of it. Pentland presents the scientific background necessary for understanding this form of communication, applies it to examples of group behavior in real organizations, and shows how by "reading" our social networks we can become more successful at pitching an idea, getting a job, or closing a deal. Using this "network intelligence" theory of social signaling, Pentland describes how we can harness the intelligence of our social network to become better managers, workers, and communicators.
---
paper_title: Stuff I've seen: a system for personal information retrieval and re-use

paper_content:

Most information retrieval technologies are designed to facilitate information discovery. However, much knowledge work involves finding and re-using previously seen information. We describe the design and evaluation of a system, called Stuff I've Seen (SIS), that facilitates information re-use. This is accomplished in two ways. First, the system provides a unified index of information that a person has seen, whether it was seen as email, web page, document, appointment, etc. Second, because the information has been seen before, rich contextual cues can be used in the search interface. The system has been used internally by more than 230 employees. We report on both qualitative and quantitative aspects of system use. Initial findings show that time and people are important retrieval cues. Users find information more easily using SIS, and use other search tools less frequently after installation.
---
paper_title: RoleNet: Movie Analysis from the Perspective of Social Networks

paper_content:

With the idea of social network analysis, we propose a novel way to analyze movie videos from the perspective of social relationships rather than audiovisual features. To appropriately describe role's relationships in movies, we devise a method to quantify relations and construct role's social networks, called RoleNet. Based on RoleNet, we are able to perform semantic analysis that goes beyond conventional feature-based approaches. In this work, social relations between roles are used to be the context information of video scenes, and leading roles and the corresponding communities can be automatically determined. The results of community identification provide new alternatives in media management and browsing. Moreover, by describing video scenes with role's context, social-relation-based story segmentation method is developed to pave a new way for this widely-studied topic. Experimental results show the effectiveness of leading role determination and community identification. We also demonstrate that the social-based story segmentation approach works much better than the conventional tempo-based method. Finally, we give extensive discussions and state that the proposed ideas provide insights into context-based video analysis.
---
paper_title: Implicit Human-Centered Tagging

paper_content:

This paper provides a general introduction to the concept of Implicit Human-Centered Tagging (IHCT) — the automatic extraction of tags from nonverbal behavioral feedback of media users. The main idea behind IHCT is that nonverbal behaviors displayed when interacting with multimedia data (e.g., facial expressions, head nods, etc.) provide information useful for improving the tag sets associated with the data. As such behaviors are displayed naturally and spontaneously, no effort is required from the users, and this is why the resulting tagging process is said to be “implicit”. Tags obtained through IHCT are expected to be more robust than tags associated with the data explicitly, at least in terms of: generality (they make sense to everybody) and statistical reliability (all tags will be sufficiently represented). The paper discusses these issues in detail and provides an overview of pioneering efforts in the field.
---
paper_title: Integrating facial expressions into user profiling for the improvement of a multimodal recommender system

paper_content:

Over the years, recommender systems have been systematically applied in both industry and academia to assist users in dealing with information overload. One of the factors that determine the performance of a recommender system is user feedback, which has been traditionally communicated through the application of explicit and implicit feedback techniques. In this paper, we propose a novel video search interface that predicts the topical relevance of a video by analysing affective aspects of user behaviour. We, furthermore, present a method for incorporating such affective features into user profiling, to facilitate the generation of meaningful recommendations, of unseen videos. Our experiment shows that multimodal interaction feature is a promising way to improve the performance of recommendation.
---
paper_title: Mobile presence and intimacy—Reshaping social actions in mobile contextual configuration

paper_content:

Abstract Mobile communication has putatively affected our time–space relationship and the co-ordination of social action by weaving co-present interactions and mediated distant exchanges into a single, seamless web. In this article, we use Goodwin's notion of contextual configuration to review, elaborate and specify these processes. Goodwin defines contextual configuration as a local, interwoven set of language and material structures that frame social production of action and meaning. We explore how the mobile context is configured in mobile phone conversations. Based on the analysis of recordings of mobile conversation in Finland and Sweden, we analyze the ways in which ordinary social actions such as invitations and offers are carried out while people are mobile. We suggest that the mobile connection introduces a special kind of relationship to semiotic resources, creating its own conditions for emerging social actions. The reformation of social actions in mobility involves the possibility of intimate connection to the ongoing activities of the distant party. The particularities of mobile social actions are discerned here through sequential analysis that opens up contextually reconfigured actions as they are revealed in the details of mobile communication. In this way, we shed light on the reformation of social actions in mobile space–time.
---
paper_title: Smartphones An Emerging Tool for Social Scientists

paper_content:

Recent developments in mobile technologies have produced a new kind of device: a programmable mobile phone, the smartphone. In this article, the authors argue that the technological and social char...
---
paper_title: ClearBoard: a seamless medium for shared drawing and conversation with eye contact

paper_content:

This paper introduces a novel shared drawing medium called ClearBoard. It realizes (1) a seamless shared drawing space and (2) eye contact to support realtime and remote collaboration by two users. We devised the key metaphor: “talking through and drawing on a transparent glass window” to design ClearBoard. A prototype of ClearBoard is implemented based on the “Drafter-Mirror” architecture. This paper first reviews previous work on shared drawing support to clarify the design goals. We then examine three methaphors that fulfill these goals. The design requirements and the two possible system architectures of ClearBoard are described. Finally, some findings gained through the experimental use of the prototype, including the feature of “gaze awareness”, are discussed.
---
paper_title: Wired for Speech: How Voice Activates and Advances the Human-Computer Relationship

paper_content:

Interfaces that talk and listen are populating computers, cars, call centers, and even home appliances and toys, but voice interfaces invariably frustrate rather than help. In Wired for Speech, Clifford Nass and Scott Brave reveal how interactive voice technologies can readily and effectively tap into the automatic responses all speech -- whether from human or machine -- evokes. Wired for Speech demonstrates that people are "voice-activated": we respond to voice technologies as we respond to actual people and behave as we would in any social situation. By leveraging this powerful finding, voice interfaces can truly emerge as the next frontier for efficient, user-friendly technology.Wired for Speech presents new theories and experiments and applies them to critical issues concerning how people interact with technology-based voices. It considers how people respond to a female voice in e-commerce (does stereotyping matter?), how a car's voice can promote safer driving (are "happy" cars better cars?), whether synthetic voices have personality and emotion (is sounding like a person always good?), whether an automated call center should apologize when it cannot understand a spoken request ("To Err is Interface; To Blame, Complex"), and much more. Nass and Brave's deep understanding of both social science and design, drawn from ten years of research at Nass's Stanford laboratory, produces results that often challenge conventional wisdom and common design practices. These insights will help designers and marketers build better interfaces, scientists construct better theories, and everyone gain better understandings of the future of the machines that speak with us.
---
paper_title: The 30-Sec Sale: Using Thin-Slice Judgments to Evaluate Sales Effectiveness

paper_content:

A successful sale depends on a customer's perception of the salesperson's personality, motivations, trustworthiness, and affect. Person perception research has shown that consistent and accurate assessments of these traits can be made based on very brief observations, or “thin slices.” Thus, examining impressions based on thin slices offers an effective approach to study how perceptions of salespeople translate into real-world results, such as sales performance and customer satisfaction. The literature on the accuracy of thin-slice judgments is briefly reviewed. Then, 2 studies are presented that investigated the predictive validity of judgments of salespeople based on thin slices of the vocal channel. Participants rated 20-sec audio clips extracted from interviews with a sample of sales managers, on variables gauging interpersonal skills, task-related skills, and anxiety. Results supported the hypothesis that observability of the rated variable is a key determinant in the criterion validity of thin-slice judgments. Implications for the use of thin-slice judgments in salesperson selection and customer satisfaction are discussed.
---
paper_title: A Conceptual Overview of the Self-Presentational Concerns and Response Tendencies of Focus Group Participants

paper_content:

Focus group respondents are often requested to perform tasks that require them to convey information about themselves. However, despite the potential for respondents to have self-presentational concerns, research on focus group productivity has virtually ignored extant scholarship on impression management. This shortcoming is addressed by presenting a conceptual overview of the effects of self-presentational concerns on focus group participation. A product of this overview is a conceptual model that posits that the amount and nature of information that people convey about themselves to others is a function of their eagerness to make desired impressions and their subjective probabilities of doing so. According to the model, when focus group participants are highly motivated to make desired impressions, they may be reluctant to present unbiased images of themselves. However, they are not likely to deceive unless they are confident in their abilities to ascertain and enact desired images. Those who are motivated to make desired impressions but are doubtful of doing so are likely to protect themselves by concealing self-relevant information or avoiding self-relevant issues. Implications of this model for research and practice are discussed.
---
paper_title: Automatic Generation of Non-verbal Behavior for Agents in Virtual Worlds: A System for Supporting Multimodal Conversations of Bots and Avatars

paper_content:

This paper presents a system capable of automatically adding gestures to an embodied virtual character processing information from a simple text input. Gestures are generated based on the analysis of linguistic and contextual information of the input text. The system is embedded in the virtual world called second life and consists of an in world object and an off world server component that handles the analysis. Either a user controlled avatar or a non user controlled character can be used to display the gestures, that are timed with speech output from an Text-to-Speech system, and so show non verbal behavior without pushing the user to manually select it.
---
paper_title: Socially intelligent robots: dimensions of human–robot interaction

paper_content:

Social intelligence in robots has a quite recent history in artificial intelligence and robotics. However, it has become increasingly apparent that social and interactive skills are necessary requirements in many application areas and contexts where robots need to interact and collaborate with other robots or humans. Research on human–robot interaction (HRI) poses many challenges regarding the nature of interactivity and ‘social behaviour’ in robot and humans. The first part of this paper addresses dimensions of HRI, discussing requirements on social skills for robots and introducing the conceptual space of HRI studies. In order to illustrate these concepts, two examples of HRI research are presented. First, research is surveyed which investigates the development of a cognitive robot companion. The aim of this work is to develop social rules for robot behaviour (a ‘robotiquette’) that is comfortable and acceptable to humans. Second, robots are discussed as possible educational or therapeutic toys for children with autism. The concept of interactive emergence in human–child interactions is highlighted. Different types of play among children are discussed in the light of their potential investigation in human–robot experiments. The paper concludes by examining different paradigms regarding ‘social relationships’ of robots and people interacting with them.
---

Title: Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing

Section 1: INTRODUCTION
Description 1: This section introduces the concept of Social Signal Processing (SSP) and its importance in bridging the gap between humans, who are inherently social, and machines, which lack social capabilities.

Section 2: NONVERBAL COMMUNICATION AND SOCIAL SIGNALS
Description 2: This section explores the history, components, and significance of nonverbal communication and how it relates to social signals.

Section 3: Toward a General Definition of Signals
Description 3: This section examines various definitions and theories of signals across multiple disciplines to develop a comprehensive understanding of what constitutes a social signal.

Section 4: Social Signals and Social Facts
Description 4: This section defines social signals and explains their roles in social interactions, social emotions, social attitudes, and social relations.

Section 5: AUTOMATIC ANALYSIS OF SOCIAL SIGNALS VIA NONVERBAL COMMUNICATION
Description 5: This section details the steps, current state-of-the-art techniques, challenges, and data resources involved in the automatic analysis of social signals through nonverbal communication.

Section 6: SYNTHESIS OF SOCIAL BEHAVIOR
Description 6: This section discusses the methodologies and challenges faced in the synthesis of social behaviors for artificial agents, including the generation of social actions, social emotions, social attitudes, and expressive speech.

Section 7: APPLICATIONS OF SSP
Description 7: This section highlights various application domains where Social Signal Processing can have significant impacts, such as multimedia indexing, mobile social interactions, human-computer interaction, marketing, and social robots.

Section 8: CONCLUSIONS
Description 8: This section summarizes the current state and future directions of Social Signal Processing, emphasizing the critical challenges and potential applications.