A Review of Temporal Aspects of Hand Gesture Analysis Applied to Discourse Analysis and Natural Conversation
---
paper_title: The recognition and comprehension of hand gestures: a review and research agenda

paper_content:

In this paper I review current and past approaches towards the use of hand gesture recognition and comprehension in human-computer interaction. I point out properties of natural coverbal gestures in human communication and identify challenges for gesture comprehension systems in three areas. The first challenge is to derive the meaning of a gesture given that its semantics is defined in three semiotic dimensions that have to be addressed differently. A second challenge is the spatial composition of gestures in imagistic spaces. Finally, a third technical challenge is the development of an integrated processing model for speech and gesture.
---
paper_title: HMM based hand gesture recognition: A review on techniques and approaches

paper_content:

Gesture is one of the most natural and expressive ways of communications between human and computer in a virtual reality system. We naturally use various gestures to express our own intentions in everyday life. Hand gesture is one of the important methods of non-verbal communication for human beings for its freer in movements and much more expressive than any other body parts. Hand gesture recognition has a number of potential applications in human-computer interaction, machine vision, virtual reality, machine control in industry, and so on. As a gesture is a continuous motion on a sequential time series, the HMMs (Hidden Markov Models) must be a prominent recognition tool. The most important thing in hand gesture recognition is what the input features are that best represent the characteristics of the moving hand gesture.This paper presents part of literature review on ongoing research and findings on different technique and approaches in gesture recognition using HMMs for vision-based approach.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: Gesture Recognition: A Survey

paper_content:

Gesture recognition pertains to recognizing meaningful expressions of motion by a human, involving the hands, arms, face, head, and/or body. It is of utmost importance in designing an intelligent and efficient human-computer interface. The applications of gesture recognition are manifold, ranging from sign language through medical rehabilitation to virtual reality. In this paper, we provide a survey on gesture recognition with particular emphasis on hand gestures and facial expressions. Applications involving hidden Markov models, particle filtering and condensation, finite-state machines, optical flow, skin color, and connectionist models are discussed in detail. Existing challenges and future research possibilities are also highlighted
---
paper_title: A survey of speech-hand gesture recognition for the development of multimodal interfaces in computer games

paper_content:

Hand gestures and speech constitute one of the most important modalities in human-to-human communication. The investigation of this phenomenon has been the focus of research about multimodal interfaces for speech and hand gesture recognition in human computer interaction(HCI). Computer games, as one of the most popular application areas of HCI, are likely to make use of this technology. Integration of speech and gesture in computer games make game play more enjoyable. This article does not intend to cover exhaustively all the issues related to multimodal interface creation in computer games. We primarily discuss some of the key issues related to multimodal interfaces for speech and hand gesture recognition and present a survey of current approaches to overcome the problems faced in this area.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Hand movement recognition for Brazilian Sign Language: A study using distance-based neural networks

paper_content:

In this paper, the vision-based hand movement recognition problem is formulated for the universe of discourse of the Brazilian Sign Language. In order to analyze this specific domain we have used the artificial neural networks models based on distance, including neural-fuzzy models. The experiments explored here show the usefulness of these models to extract helpful knowledge about the classes of movements and to support the project of adaptative recognizer modules for Libras-oriented computational tools. Using artificial neural networks architectures - Self Organizing Maps and (Fuzzy) Learning Vector Quantization, it was possible to understand the data space and to build models able to recognize hand movements performed for one or more than one specific Libras users.
---
paper_title: Multimodal human discourse: gesture and speech

paper_content:

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues.
---
paper_title: Catchments, prosody and discourse

paper_content:

We present our work on the temporal integration of hierarchies of communicative actions: kinesic, prosodic and discursive. We use the device of the ‘catchment’ as the locus around which this integration proceeds. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. First, we process the video data to obtain the motion traces of both of the subject’s hands using the vector coherence mapping algorithm. We code the gestures to identify the catchments. We recover discourse purposes utilizing a system of guided questions. Finally, we define prosody in terms of the ToBI system. The results of these analyses are compared against the computed motion traces to identify the cues accessible in the gestural and audio data that correlate well with the psycholinguistic analyses. The results show that motion, prosody and discourse structure are integrated at each moment of speaking.The electronic edition of this article includes audio-visual data.
---
paper_title: Psycholinguistics: Central Topics

paper_content:

Acknowledgements Preface 1 Introduction - how to study language understanding 2 The contribution of linguistics 3 Recognizing words 4 Parsing - the computation of syntactic structure 5 Introduction to the concept of meaning 6 Word meaning 7 Understanding discourse and text 8 The structure of the language processor 9 Language production and its relation to comprehension 10 Overview and future directions References Name index Subject index
---
paper_title: Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders

paper_content:

The previous literature has suggested that the hand movement in co-speech gestures and signs consists of a series of phases with qualitatively different dynamic characteristics. In this paper, we propose a syntagmatic rule system for movement phases that applies to both co-speech gestures and signs. Descriptive criteria for the rule system were developed for the analysis video-recorded continuous production of signs and gesture. It involves segmenting a stream of body movement into phases and identifying different phase types. Two human coders used the criteria to analyze signs and cospeech gestures that are produced in natural discourse. It was found that the criteria yielded good inter-coder reliability. These criteria can be used for the technology of automatic recognition of signs and co-speech gestures in order to segment continuous production and identify the potentially meaningbearing phase.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: The recognition and comprehension of hand gestures: a review and research agenda

paper_content:

In this paper I review current and past approaches towards the use of hand gesture recognition and comprehension in human-computer interaction. I point out properties of natural coverbal gestures in human communication and identify challenges for gesture comprehension systems in three areas. The first challenge is to derive the meaning of a gesture given that its semantics is defined in three semiotic dimensions that have to be addressed differently. A second challenge is the spatial composition of gestures in imagistic spaces. Finally, a third technical challenge is the development of an integrated processing model for speech and gesture.
---
paper_title: Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders

paper_content:

The previous literature has suggested that the hand movement in co-speech gestures and signs consists of a series of phases with qualitatively different dynamic characteristics. In this paper, we propose a syntagmatic rule system for movement phases that applies to both co-speech gestures and signs. Descriptive criteria for the rule system were developed for the analysis video-recorded continuous production of signs and gesture. It involves segmenting a stream of body movement into phases and identifying different phase types. Two human coders used the criteria to analyze signs and cospeech gestures that are produced in natural discourse. It was found that the criteria yielded good inter-coder reliability. These criteria can be used for the technology of automatic recognition of signs and co-speech gestures in order to segment continuous production and identify the potentially meaningbearing phase.
---
paper_title: HMM based hand gesture recognition: A review on techniques and approaches

paper_content:

Gesture is one of the most natural and expressive ways of communications between human and computer in a virtual reality system. We naturally use various gestures to express our own intentions in everyday life. Hand gesture is one of the important methods of non-verbal communication for human beings for its freer in movements and much more expressive than any other body parts. Hand gesture recognition has a number of potential applications in human-computer interaction, machine vision, virtual reality, machine control in industry, and so on. As a gesture is a continuous motion on a sequential time series, the HMMs (Hidden Markov Models) must be a prominent recognition tool. The most important thing in hand gesture recognition is what the input features are that best represent the characteristics of the moving hand gesture.This paper presents part of literature review on ongoing research and findings on different technique and approaches in gesture recognition using HMMs for vision-based approach.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: The recognition and comprehension of hand gestures: a review and research agenda

paper_content:

In this paper I review current and past approaches towards the use of hand gesture recognition and comprehension in human-computer interaction. I point out properties of natural coverbal gestures in human communication and identify challenges for gesture comprehension systems in three areas. The first challenge is to derive the meaning of a gesture given that its semantics is defined in three semiotic dimensions that have to be addressed differently. A second challenge is the spatial composition of gestures in imagistic spaces. Finally, a third technical challenge is the development of an integrated processing model for speech and gesture.
---
paper_title: A survey of speech-hand gesture recognition for the development of multimodal interfaces in computer games

paper_content:

Hand gestures and speech constitute one of the most important modalities in human-to-human communication. The investigation of this phenomenon has been the focus of research about multimodal interfaces for speech and hand gesture recognition in human computer interaction(HCI). Computer games, as one of the most popular application areas of HCI, are likely to make use of this technology. Integration of speech and gesture in computer games make game play more enjoyable. This article does not intend to cover exhaustively all the issues related to multimodal interface creation in computer games. We primarily discuss some of the key issues related to multimodal interfaces for speech and hand gesture recognition and present a survey of current approaches to overcome the problems faced in this area.
---
paper_title: Hand movement recognition for Brazilian Sign Language: A study using distance-based neural networks

paper_content:

In this paper, the vision-based hand movement recognition problem is formulated for the universe of discourse of the Brazilian Sign Language. In order to analyze this specific domain we have used the artificial neural networks models based on distance, including neural-fuzzy models. The experiments explored here show the usefulness of these models to extract helpful knowledge about the classes of movements and to support the project of adaptative recognizer modules for Libras-oriented computational tools. Using artificial neural networks architectures - Self Organizing Maps and (Fuzzy) Learning Vector Quantization, it was possible to understand the data space and to build models able to recognize hand movements performed for one or more than one specific Libras users.
---
paper_title: Segmentation and Recognition of Continuous Gestures

paper_content:

A novel method is introduced to segment and recognize time-varying human gestures from continuous video streams. Motion is represented by a 3D spatio-temporal surface based upon the evolution of a contour over time. The warping paths between the input signal and a set of gesture models are obtained using continuous dynamic programming and the boundary of a gesture is located by analyzing all possible gesture candidates during a specific period of time. Correlation and mutual information are employed to select the best candidate when more than one gesture is recognized at the same time period. The system has been implemented and tested on continuous gesture sequences containing 8 different gestures performed by 4 subjects. The results demonstrate that the proposed method is very effective, achieving a recognition rate of 95.9%.
---
paper_title: Automated Posture Segmentation in Continuous Finger Spelling Recognition

paper_content:

Recognizing continuous finger spelling plays an important role in understanding sign language. There are two major phases in recognizing continuous finger spelling, which are posture segmentation and posture recognition. In the former, a continuous gesture sequence is decomposed into segments, which are then used for the latter to indentify corresponding characters. Among all the segments, beside valid postures corresponding to characters, there are also many movement epentheses, which appear between pairs of postures to move the hands from the end of one posture to the beginning of the next. In this paper, we propose a framework to split a continuous movement sequence into segments as well as to identify valid postures and movement epentheses. By using the velocity and signing rate based filter, we can obtain very good result with both high recall and precision rate.
---
paper_title: BoVDW: Bag-of-Visual-and-Depth-Words for gesture recognition

paper_content:

We present a Bag-of-Visual-and-Depth-Words (BoVDW) model for gesture recognition, an extension of the Bag-of-Visual-Words (BoVW) model, that benefits from the multimodal fusion of visual and depth features. State-of-the-art RGB and depth features, including a new proposed depth descriptor, are analysed and combined in a late fusion fashion. The method is integrated in a continuous gesture recognition pipeline, where Dynamic Time Warping (DTW) algorithm is used to perform prior segmentation of gestures. Results of the method in public data sets, within our gesture recognition pipeline, show better performance in comparison to a standard BoVW model.
---
paper_title: A Compositional Model for Gesture Definition

paper_content:

The description of a gesture requires temporal analysis of values generated by input sensors and does not fit well the observer pattern traditionally used by frameworks to handle user input. The current solution is to embed particular gesture-based interactions, such as pinch-to-zoom, into frameworks by notifying when a whole gesture is detected. This approach suffers from a lack of flexibility unless the programmer performs explicit temporal analysis of raw sensors data. This paper proposes a compositional, declarative meta-model for gestures definition based on Petri Nets. Basic traits are used as building blocks for defining gestures; each one notifies the change of a feature value. A complex gesture is defined by the composition of other sub-gestures using a set of operators. The user interface behaviour can be associated to the recognition of the whole gesture or to any other sub-component, addressing the problem of granularity for the notification events. The meta-model can be instantiated for different gesture recognition supports and its definition has been validated through a proof of concept library. Sample applications have been developed for supporting multitouch gestures on iOS and full body gestures with Microsoft Kinect.
---
paper_title: Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders

paper_content:

The previous literature has suggested that the hand movement in co-speech gestures and signs consists of a series of phases with qualitatively different dynamic characteristics. In this paper, we propose a syntagmatic rule system for movement phases that applies to both co-speech gestures and signs. Descriptive criteria for the rule system were developed for the analysis video-recorded continuous production of signs and gesture. It involves segmenting a stream of body movement into phases and identifying different phase types. Two human coders used the criteria to analyze signs and cospeech gestures that are produced in natural discourse. It was found that the criteria yielded good inter-coder reliability. These criteria can be used for the technology of automatic recognition of signs and co-speech gestures in order to segment continuous production and identify the potentially meaningbearing phase.
---
paper_title: Continuous Gesture Recognition using a Sparse Bayesian Classifier

paper_content:

An approach to recognise and segment 9 elementary gestures from a video input is proposed and it can be applied to continuous sign recognition. An isolated gesture is recognised by first converting a portion of video into a motion gradient orientation image and then classifying it into one of the 9 gestures by a sparse Bayesian classifier. The portion of video used is decided by using a sampling technique based on CONDENSATION framework. By doing so, gestures can be segmented from the video in a probabilistic manner. Experiments show that the proposed method can achieve accuracy around 90% in both isolated and continuous gesture recognition without using special equipment such as glove devices and the system can run in real-time.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: Temporal series recognition using a new neural network structure T-CombNET

paper_content:

We present a new neural network structure dedicated to the temporal data series recognition, called T-CombNET (Temporal CombNET). The T-CombNET is a modified and improved version of the CombNET-II structure, which was designed to deal with very large vocabulary character recognition. We incorporated a time normalization stage and used recurrent neural networks to enable CombNET-II to do the temporal analysis. Then we applied this new model T-CombNET to the hand gesture recognition task. The preliminary results show that the proposed structure is very efficient, obtaining a recognition rate of 99.41% in visual based kana hand alphabet recognition experiments.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Improved dynamic gesture segmentation for Thai sign language translation

paper_content:

Sign languages normally make use of both static and dynamic gestures to achieve communication goal. In automated continuous gesture recognition, one of the problems found is how to separate a transitional gesture from a meaningful dynamic gesture. In addition, a variation in speed, absolute position and orientation of a dynamic gesture makes an automatic recognition task nontrivial. This paper proposes a segmentation method that alleviates these problems. The method is based on the fact that many (meaningful) dynamic gestures as appeared in Thai and other sign languages, are performed in an approximately (quasi) periodic manner. Therefore, such dynamic gestures can be distinguished from a transitional gesture by means of Fourier analysis, performed on samples (discrete signal) of hand shapes, positions and orientation. The analysis, however, requires a good choice of data sample windowing. This paper describes a preprocessing method for estimation of the dynamic gesture data windowing period. The proposed method helps improving the accuracy of segmenting hand shape and location data samples into static, periodic and nonperiodic gestures. In addition, accuracy of the feature extracted from the Fourier analysis of a dynamic gesture signal is improved, thus increasing the recognition rate. Experimental results performed on a real data are included.
---
paper_title: Slideshow: Gesture-aware PPT presentation

paper_content:

Traditional keyboard and mouse based presentation prevents lecturers from interacting with the audiences freely and closely. In this paper, we propose a gesture-aware presentation tool named SlideShow to liberate lecturers from physical space constraints and make human-computer interaction more natural and convenient. In our system, gesture data is obtained by a handle controller with 3-axis accelerometer and gyro and transmitted to host-side through bluetooth, then we use Bayesian change point detection to segment continuous gesture series and HMM to recognize the gesture. In consequence Slideshow could carry out the corresponding operations on PowerPoint(PPT) to make a presentation, and operation states can be switched automatically and intelligently during the presentation. Both the experimental and testing results show our approach is practical, useful and convenient.
---
paper_title: Image-based gesture recognition with support vector machines

paper_content:

Recent advances in various display and virtual technologies, coupled with an explosion in available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities, among which gesture recognition is undoubtedly the most grammatically structured and complex. However, despite the abundance of novel interaction devices, the naturalness and efficiency of HCI has remained low. This is due in particular to the lack of robust sensory data interpretation techniques. To address the task of gesture recognition, this dissertation establishes novel probabilistic approaches based on support vector machines (SVM). Of special concern in this dissertation are the shapes of contact images on a multi-touch input device for both 2D and 3D. Five main topics are covered in this work. ::: The first topic deals with the hand pose recognition problem. To perform classification of different gestures, a recognition system must attempt to leverage between class variations (semantically varying gestures), while accommodating potentially large within-class variations (different hand poses to perform certain gestures). For recognition of gestures, a sequence of hand shapes should be recognized. We present a novel shape recognition approach using Active Shape Model (ASM) based matching and SVM based classification. Firstly, a set of correspondences between the reference shape and query image are identified through ASM. Next, a dissimilarity measure is created to measure how well any correspondence in the set aligns the reference shape and candidate shape in the query image. Finally, SVM classification is employed to search through the set to find the best match from the kernel defined by the dissimilarity measure above. Results presented show better recognition results than conventional segmentation and template matching methods. ::: In the second topic, dynamic time alignment (DTA) based SVM gesture recognition is addressed. In particular, the proposed method combines DTA and SVM by establishing a new kernel. The gesture data is first projected into a common eigenspace formed by principal component analysis (PCA) and a distance measure is derived from the DTA. By incorporating DTA in the kernel function, general classification problems with variable-sized sequential data can be handled. ::: In the third topic, a C++ based gesture recognition application for the multi-touchpad is implemented. It uses the proposed gesture classification method along with a recursive neural networks approach to recognize definable gestures in real time, then runs an associated command. This application can further enable users with different disabilities or preferences to custom define gestures and enhance the functionality of the multi-touchpad. ::: Fourthly, an SVM-based classification method that uses the DTW to measure the similarity score is presented. The key contribution of this approach is the extension of trajectory based approaches to handle shape information, thereby enabling the expansion of the system's gesture vocabulary. It consists of two steps: converting a given set of frames into fixed-length vectors and training an SVM from the vectorized manifolds. Using shape information not only yields discrimination among various gestures, but also enables gestures that cannot be characterized solely based on their motion information to be classified, thus boosting overall recognition scores. ::: Finally, a computer vision based gesture command and communication system is developed. This system performs two major tasks: the first is to utilize the 3D traces of laser pointing devices as input to perform common keyboard and mouse control; the second is supplement free continuous gesture recognition, i.e., data gloves or other assistive devices are not necessary for 3D gestures recognition. As a result, the gesture can be used as a text entry system in wearable computers or mobile communication devices, though the recognition rate is lower thanthe approaches with the assistive tools. The purpose of this system is to develop new perceptual interfaces for human computer interaction based on visual input captured by computer vision systems, and to investigate how such interfaces can complement or replace traditional interfaces. ::: Original contributions of this work span the areas of SVMs and interpretation of computer sensory inputs, such as gestures for advanced HCI. In particular, we have addressed the following important issues: (1) ASM base kernels for shape recognition. (2) DTA based sequence kernels for gesture classification. (3) Recurrent neural networks (RNN). (4) Exploration of a customizable HCI. (5) Computer vision based 3D gesture recognition algorithms and system.
---
paper_title: Visual recognition of aircraft marshalling signals using gesture phase analysis

paper_content:

Visual gesture recognition is one of the main areas of research in human-computer and human-robot interfaces. In this paper we present a novel visual gesture recognition method for aircraft marshalling signals. To capture hand motion information, we used a color-based tracking algorithm with an adaptive window for each frame. A feature selection algorithm was used to classify the motion information into four different gesture phases. By using the gesture phase information, we built the gesture model, which consisted of a symbol sequence and a number of probabilities. Each gesture model was learned from the longest common subsequence (LCS) of a set of symbol sequences and the probability of the symbols. A similarity measure using the proposed gesture model is presented that combines the deterministic and probabilistic matching scores. In the experiments we show the efficiency and accuracy of the proposed method.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Capturing Expressive and Indicative Qualities of Conducting Gesture: An Application of Temporal Expectancy Models

paper_content:

Many event sequences in everyday human movement exhibit temporal structure: for instance, footsteps in walking, the striking of balls in a tennis match, the movements of a dancer set to rhythmic music, and the gestures of an orchestra conductor. These events generate prior expectancies regarding the occurrence of future events. Moreover, these expectancies play a critical role in conveying expressive qualities and communicative intent through the movement; thus they are of considerable interest in musical control contexts. To this end, we introduce a novel Bayesian framework which we call the temporal expectancy modeland use it to develop an analysis tool for capturing expressiveand indicativequalities of the conducting gesture based on temporal expectancies. The temporal expectancy model is a general dynamic Bayesian network (DBN) that can be used to encode prior knowledge regarding temporal structure to improve event segmentation. The conducting analysis tool infers beat and tempo, which are indicative and articulation which is expressive, as well as temporal expectancies regarding beat (ictusand preparationinstances) from conducting gesture. Experimental results using our analysis framework reveal a very strong correlation in how significantly the preparation expectancy builds up for staccato vs legato articulation, which bolsters the case for temporal expectancy as cognitive model for event anticipation, and as a key factor in the communication of expressive qualities of conducting gesture. Our system operates on data obtained from a marker based motion capture system, but can be easily adapted for more affordable technologies like video camera arrays.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Realtime online adaptive gesture recognition

paper_content:

We introduce an online adaptive algorithm for learning gesture models. By learning gesture models in an online fashion, the gesture recognition process is made more robust, and the need to train on a large training ensemble is obviated. Hidden Markov models are used to represent the spatial and temporal structure of the gesture. The usual output probability distributions-typically representing appearance-are trained at runtime exploiting the temporal structure (Markov model) that is either trained off-line or is explicitly hand-coded. In the early stages of runtime adaptation, contextural information derived from the application is used to bias the expectation as to which Markov state the system is in at any given time. We describe the Watch and Learn system, a computer vision system which is able to learn simple gestures online for interactive control.
---
paper_title: Modeling gesture salience as a hidden variable for coreference resolution and keyframe extraction

paper_content:

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and noncommunicative hand motions may confuse natural language processing (NLP) and impede learning. People have little difficulty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: Discourse topic and gestural form

paper_content:

Coverbal gesture provides a channel for the visual expression of ideas. While some gestural emblems have culturally predefined forms (e.g., "thumbs up"), the relationship between gesture and meaning is, in general, not conventionalized. It is natural to ask whether such gestures can be interpreted in a speaker-independent way, or whether gestural form is determined by the speaker's idiosyncratic view of the discourse topic. We address this question using an audiovisual dataset across multiple speakers and topics. Our analysis employs a hierarchical Bayesian author-topic model, in which gestural patterns are stochastically generated by a mixture of speaker-specific and topic-specific priors. These gestural patterns are characterized using automatically-extracted visual features, based on spatio-temporal interest points. This framework detects significant cross-speaker patterns in gesture that are governed by the discourse topic, suggesting that even unstructured gesticulation can be interpreted across speakers. In addition, the success of this approach shows that the semantic characteristics of gesture can be detected via a low-level, interest point representation.
---
paper_title: Multimodal model integration for sentence unit detection

paper_content:

In this paper, we adopt a direct modeling approach to utilize conversational gesture cues in detecting sentence boundaries, called SUs, in video taped conversations. We treat the detection of SUs as a classification task such that for each inter-word boundary, the classifier decides whether there is an SU boundary or not. In addition to gesture cues, we also utilize prosody and lexical knowledge sources. In this first investigation, we find that gesture features complement the prosodic and lexical knowledge sources for this task. By using all of the knowledge sources, the model is able to achieve the lowest overall SU detection error rate.
---
paper_title: Multimodal human discourse: gesture and speech

paper_content:

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues.
---
paper_title: Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders

paper_content:

The previous literature has suggested that the hand movement in co-speech gestures and signs consists of a series of phases with qualitatively different dynamic characteristics. In this paper, we propose a syntagmatic rule system for movement phases that applies to both co-speech gestures and signs. Descriptive criteria for the rule system were developed for the analysis video-recorded continuous production of signs and gesture. It involves segmenting a stream of body movement into phases and identifying different phase types. Two human coders used the criteria to analyze signs and cospeech gestures that are produced in natural discourse. It was found that the criteria yielded good inter-coder reliability. These criteria can be used for the technology of automatic recognition of signs and co-speech gestures in order to segment continuous production and identify the potentially meaningbearing phase.
---
paper_title: Hand motion gestural oscillations and multimodal discourse

paper_content:

To develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion oscillatory gesture detection in natural speech and conversation. First, the hand motion trajectory signals are extracted from video. Second, a wavelet analysis based approach is presented to process the signals. In this approach, wavelet ridges are extracted from the responses of wavelet analysis for the hand motion trajectory signals, which can be used to characterize frequency properties of the hand motion signals. The hand motion oscillatory gestures can be extracted from these frequency properties. Finally, we relate the hand motion oscillatory gestures to the phases of speech and multimodal discourse analysis.We demonstrate the efficacy of the system on a real discourse dataset in which a subject described her action plan to an interlocutor. We extracted the oscillatory gestures from the x, y and z motion traces of both hands. We further demonstrate the power of gestural oscillation detection as a key to unlock the structure of the underlying discourse.
---
paper_title: The catchment feature model: A device for multimodal fusion and a bridge between signal and sense

paper_content:

The catchment feature model addresses two questions in the field of multimodal interaction: how we bridge video and audio processing with the realities of human multimodal communication, and how information from the different modes may be fused. We argue from a detailed literature review that gestural research has clustered around manipulative and semaphoric use of the hands, motivate the catchment feature model psycholinguistic research, and present the model. In contrast to "whole gesture" recognition, the catchment feature model applies a feature decomposition approach that facilitates cross-modal fusion at the level of discourse planning and conceptualization. We present our experimental framework for catchment feature-based research, cite three concrete examples of catchment features, and propose new directions of multimodal research based on the model.
---
paper_title: Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis

paper_content:

Gesture and speech are co-expressive and complementary channels of a single human language system. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. We investigate the role of oscillatory/cyclical hand motions in `carrying' this image content. We present our work on the extraction of hand motion oscillation frequencies of gestures that accompany speech. The key challenges are that such motions are characterized by non-stationary oscillations, and multiple frequencies may be simultaneously extant. Also, the duration of the oscillations may be extended over very few cycles. We apply the windowed Fourier transform and wavelet transform to detect and extract gesticulatory oscillations. We tested these against synthetic signals (stationary and non-stationary) and real data sequences of gesticulatory hand movements in natural discourse. Our results show that both filters functioned well for the synthetic signals. For the real data, the wavelet bandpass filter bank is better for detecting and extracting hand gesture oscillations. We relate the hand motion oscillatory gestures detected by wavelet analysis to speech in natural conversation and apply to multimodal language analysis. We demonstrate the ability of our algorithm to extract gesticulatory oscillations and show how oscillatory gestures reveal portions of the multimodal discourse structure.
---
paper_title: Gesture in Automatic Discourse Processing

paper_content:

Computers cannot fully understand spoken language without access to the wide range of modalities that accompany speech. This thesis addresses the particularly expressive modality of hand gesture, and focuses on building structured statistical models at the intersection of speech, vision, and meaning. ::: My approach is distinguished in two key respects. First, gestural patterns are leveraged to discover parallel structures in the meaning of the associated speech. This differs from prior work that attempted to interpret individual gestures directly, an approach that was prone to a lack of generality across speakers. Second, I present novel, structured statistical models for multimodal language processing, which enable learning about gesture in its linguistic context, rather than in the abstract. ::: These ideas find successful application in a variety of language processing tasks: resolving ambiguous noun phrases, segmenting speech into topics, and producing keyframe summaries of spoken language. In all three cases, the addition of gestural features—extracted automatically from video—yields significantly improved performance over a state-of-the-art text-only alternative. This marks the first demonstration that hand gesture improves automatic discourse processing. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)
---
paper_title: Hand gesture symmetric behavior detection and analysis in natural conversation

paper_content:

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: Gesture patterns during speech repairs

paper_content:

Speech and gesture are two primary modes used in natural human communication; hence, they are important inputs for a multimodal interface to process. One of the challenges for multimodal interfaces is to accurately recognize the words in spontaneous speech. This is partly due to the presence of speech repairs, which seriously degrade the accuracy of current speech recognition systems. Based on the assumption that speech and gesture arise from the same thought process, we would expect to find patterns of gesture that co-occur with speech repairs that can be exploited by a multimodal processing system to more effectively process spontaneous speech. To evaluate this hypothesis, we have conducted a measurement study of gesture and speech repair data extracted from videotapes of natural dialogs. Although we have found that gestures do not always co-occur with speech repairs, we observed that modification gesture patterns have a high correlation with content replacement speech repairs, but rarely occur with content repetitions. These results suggest that gesture patterns can help us to classify different types of speech repairs in order to correct them more accurately.
---
paper_title: Segmentation and Recognition of Continuous Gestures

paper_content:

A novel method is introduced to segment and recognize time-varying human gestures from continuous video streams. Motion is represented by a 3D spatio-temporal surface based upon the evolution of a contour over time. The warping paths between the input signal and a set of gesture models are obtained using continuous dynamic programming and the boundary of a gesture is located by analyzing all possible gesture candidates during a specific period of time. Correlation and mutual information are employed to select the best candidate when more than one gesture is recognized at the same time period. The system has been implemented and tested on continuous gesture sequences containing 8 different gestures performed by 4 subjects. The results demonstrate that the proposed method is very effective, achieving a recognition rate of 95.9%.
---
paper_title: Discourse topic and gestural form

paper_content:

Coverbal gesture provides a channel for the visual expression of ideas. While some gestural emblems have culturally predefined forms (e.g., "thumbs up"), the relationship between gesture and meaning is, in general, not conventionalized. It is natural to ask whether such gestures can be interpreted in a speaker-independent way, or whether gestural form is determined by the speaker's idiosyncratic view of the discourse topic. We address this question using an audiovisual dataset across multiple speakers and topics. Our analysis employs a hierarchical Bayesian author-topic model, in which gestural patterns are stochastically generated by a mixture of speaker-specific and topic-specific priors. These gestural patterns are characterized using automatically-extracted visual features, based on spatio-temporal interest points. This framework detects significant cross-speaker patterns in gesture that are governed by the discourse topic, suggesting that even unstructured gesticulation can be interpreted across speakers. In addition, the success of this approach shows that the semantic characteristics of gesture can be detected via a low-level, interest point representation.
---
paper_title: Automated Posture Segmentation in Continuous Finger Spelling Recognition

paper_content:

Recognizing continuous finger spelling plays an important role in understanding sign language. There are two major phases in recognizing continuous finger spelling, which are posture segmentation and posture recognition. In the former, a continuous gesture sequence is decomposed into segments, which are then used for the latter to indentify corresponding characters. Among all the segments, beside valid postures corresponding to characters, there are also many movement epentheses, which appear between pairs of postures to move the hands from the end of one posture to the beginning of the next. In this paper, we propose a framework to split a continuous movement sequence into segments as well as to identify valid postures and movement epentheses. By using the velocity and signing rate based filter, we can obtain very good result with both high recall and precision rate.
---
paper_title: Multimodal model integration for sentence unit detection

paper_content:

In this paper, we adopt a direct modeling approach to utilize conversational gesture cues in detecting sentence boundaries, called SUs, in video taped conversations. We treat the detection of SUs as a classification task such that for each inter-word boundary, the classifier decides whether there is an SU boundary or not. In addition to gesture cues, we also utilize prosody and lexical knowledge sources. In this first investigation, we find that gesture features complement the prosodic and lexical knowledge sources for this task. By using all of the knowledge sources, the model is able to achieve the lowest overall SU detection error rate.
---
paper_title: A Compositional Model for Gesture Definition

paper_content:

The description of a gesture requires temporal analysis of values generated by input sensors and does not fit well the observer pattern traditionally used by frameworks to handle user input. The current solution is to embed particular gesture-based interactions, such as pinch-to-zoom, into frameworks by notifying when a whole gesture is detected. This approach suffers from a lack of flexibility unless the programmer performs explicit temporal analysis of raw sensors data. This paper proposes a compositional, declarative meta-model for gestures definition based on Petri Nets. Basic traits are used as building blocks for defining gestures; each one notifies the change of a feature value. A complex gesture is defined by the composition of other sub-gestures using a set of operators. The user interface behaviour can be associated to the recognition of the whole gesture or to any other sub-component, addressing the problem of granularity for the notification events. The meta-model can be instantiated for different gesture recognition supports and its definition has been validated through a proof of concept library. Sample applications have been developed for supporting multitouch gestures on iOS and full body gestures with Microsoft Kinect.
---
paper_title: Multimodal human discourse: gesture and speech

paper_content:

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues.
---
paper_title: Continuous Gesture Recognition using a Sparse Bayesian Classifier

paper_content:

An approach to recognise and segment 9 elementary gestures from a video input is proposed and it can be applied to continuous sign recognition. An isolated gesture is recognised by first converting a portion of video into a motion gradient orientation image and then classifying it into one of the 9 gestures by a sparse Bayesian classifier. The portion of video used is decided by using a sampling technique based on CONDENSATION framework. By doing so, gestures can be segmented from the video in a probabilistic manner. Experiments show that the proposed method can achieve accuracy around 90% in both isolated and continuous gesture recognition without using special equipment such as glove devices and the system can run in real-time.
---
paper_title: The catchment feature model: A device for multimodal fusion and a bridge between signal and sense

paper_content:

The catchment feature model addresses two questions in the field of multimodal interaction: how we bridge video and audio processing with the realities of human multimodal communication, and how information from the different modes may be fused. We argue from a detailed literature review that gestural research has clustered around manipulative and semaphoric use of the hands, motivate the catchment feature model psycholinguistic research, and present the model. In contrast to "whole gesture" recognition, the catchment feature model applies a feature decomposition approach that facilitates cross-modal fusion at the level of discourse planning and conceptualization. We present our experimental framework for catchment feature-based research, cite three concrete examples of catchment features, and propose new directions of multimodal research based on the model.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: Temporal series recognition using a new neural network structure T-CombNET

paper_content:

We present a new neural network structure dedicated to the temporal data series recognition, called T-CombNET (Temporal CombNET). The T-CombNET is a modified and improved version of the CombNET-II structure, which was designed to deal with very large vocabulary character recognition. We incorporated a time normalization stage and used recurrent neural networks to enable CombNET-II to do the temporal analysis. Then we applied this new model T-CombNET to the hand gesture recognition task. The preliminary results show that the proposed structure is very efficient, obtaining a recognition rate of 99.41% in visual based kana hand alphabet recognition experiments.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Capturing Expressive and Indicative Qualities of Conducting Gesture: An Application of Temporal Expectancy Models

paper_content:

Many event sequences in everyday human movement exhibit temporal structure: for instance, footsteps in walking, the striking of balls in a tennis match, the movements of a dancer set to rhythmic music, and the gestures of an orchestra conductor. These events generate prior expectancies regarding the occurrence of future events. Moreover, these expectancies play a critical role in conveying expressive qualities and communicative intent through the movement; thus they are of considerable interest in musical control contexts. To this end, we introduce a novel Bayesian framework which we call the temporal expectancy modeland use it to develop an analysis tool for capturing expressiveand indicativequalities of the conducting gesture based on temporal expectancies. The temporal expectancy model is a general dynamic Bayesian network (DBN) that can be used to encode prior knowledge regarding temporal structure to improve event segmentation. The conducting analysis tool infers beat and tempo, which are indicative and articulation which is expressive, as well as temporal expectancies regarding beat (ictusand preparationinstances) from conducting gesture. Experimental results using our analysis framework reveal a very strong correlation in how significantly the preparation expectancy builds up for staccato vs legato articulation, which bolsters the case for temporal expectancy as cognitive model for event anticipation, and as a key factor in the communication of expressive qualities of conducting gesture. Our system operates on data obtained from a marker based motion capture system, but can be easily adapted for more affordable technologies like video camera arrays.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis

paper_content:

Gesture and speech are co-expressive and complementary channels of a single human language system. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. We investigate the role of oscillatory/cyclical hand motions in `carrying' this image content. We present our work on the extraction of hand motion oscillation frequencies of gestures that accompany speech. The key challenges are that such motions are characterized by non-stationary oscillations, and multiple frequencies may be simultaneously extant. Also, the duration of the oscillations may be extended over very few cycles. We apply the windowed Fourier transform and wavelet transform to detect and extract gesticulatory oscillations. We tested these against synthetic signals (stationary and non-stationary) and real data sequences of gesticulatory hand movements in natural discourse. Our results show that both filters functioned well for the synthetic signals. For the real data, the wavelet bandpass filter bank is better for detecting and extracting hand gesture oscillations. We relate the hand motion oscillatory gestures detected by wavelet analysis to speech in natural conversation and apply to multimodal language analysis. We demonstrate the ability of our algorithm to extract gesticulatory oscillations and show how oscillatory gestures reveal portions of the multimodal discourse structure.
---
paper_title: Gesture in Automatic Discourse Processing

paper_content:

Computers cannot fully understand spoken language without access to the wide range of modalities that accompany speech. This thesis addresses the particularly expressive modality of hand gesture, and focuses on building structured statistical models at the intersection of speech, vision, and meaning. ::: My approach is distinguished in two key respects. First, gestural patterns are leveraged to discover parallel structures in the meaning of the associated speech. This differs from prior work that attempted to interpret individual gestures directly, an approach that was prone to a lack of generality across speakers. Second, I present novel, structured statistical models for multimodal language processing, which enable learning about gesture in its linguistic context, rather than in the abstract. ::: These ideas find successful application in a variety of language processing tasks: resolving ambiguous noun phrases, segmenting speech into topics, and producing keyframe summaries of spoken language. In all three cases, the addition of gestural features—extracted automatically from video—yields significantly improved performance over a state-of-the-art text-only alternative. This marks the first demonstration that hand gesture improves automatic discourse processing. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)
---
paper_title: Hand gesture symmetric behavior detection and analysis in natural conversation

paper_content:

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.
---
paper_title: Improved dynamic gesture segmentation for Thai sign language translation

paper_content:

Sign languages normally make use of both static and dynamic gestures to achieve communication goal. In automated continuous gesture recognition, one of the problems found is how to separate a transitional gesture from a meaningful dynamic gesture. In addition, a variation in speed, absolute position and orientation of a dynamic gesture makes an automatic recognition task nontrivial. This paper proposes a segmentation method that alleviates these problems. The method is based on the fact that many (meaningful) dynamic gestures as appeared in Thai and other sign languages, are performed in an approximately (quasi) periodic manner. Therefore, such dynamic gestures can be distinguished from a transitional gesture by means of Fourier analysis, performed on samples (discrete signal) of hand shapes, positions and orientation. The analysis, however, requires a good choice of data sample windowing. This paper describes a preprocessing method for estimation of the dynamic gesture data windowing period. The proposed method helps improving the accuracy of segmenting hand shape and location data samples into static, periodic and nonperiodic gestures. In addition, accuracy of the feature extracted from the Fourier analysis of a dynamic gesture signal is improved, thus increasing the recognition rate. Experimental results performed on a real data are included.
---
paper_title: Slideshow: Gesture-aware PPT presentation

paper_content:

Traditional keyboard and mouse based presentation prevents lecturers from interacting with the audiences freely and closely. In this paper, we propose a gesture-aware presentation tool named SlideShow to liberate lecturers from physical space constraints and make human-computer interaction more natural and convenient. In our system, gesture data is obtained by a handle controller with 3-axis accelerometer and gyro and transmitted to host-side through bluetooth, then we use Bayesian change point detection to segment continuous gesture series and HMM to recognize the gesture. In consequence Slideshow could carry out the corresponding operations on PowerPoint(PPT) to make a presentation, and operation states can be switched automatically and intelligently during the presentation. Both the experimental and testing results show our approach is practical, useful and convenient.
---
paper_title: Image-based gesture recognition with support vector machines

paper_content:

Recent advances in various display and virtual technologies, coupled with an explosion in available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities, among which gesture recognition is undoubtedly the most grammatically structured and complex. However, despite the abundance of novel interaction devices, the naturalness and efficiency of HCI has remained low. This is due in particular to the lack of robust sensory data interpretation techniques. To address the task of gesture recognition, this dissertation establishes novel probabilistic approaches based on support vector machines (SVM). Of special concern in this dissertation are the shapes of contact images on a multi-touch input device for both 2D and 3D. Five main topics are covered in this work. ::: The first topic deals with the hand pose recognition problem. To perform classification of different gestures, a recognition system must attempt to leverage between class variations (semantically varying gestures), while accommodating potentially large within-class variations (different hand poses to perform certain gestures). For recognition of gestures, a sequence of hand shapes should be recognized. We present a novel shape recognition approach using Active Shape Model (ASM) based matching and SVM based classification. Firstly, a set of correspondences between the reference shape and query image are identified through ASM. Next, a dissimilarity measure is created to measure how well any correspondence in the set aligns the reference shape and candidate shape in the query image. Finally, SVM classification is employed to search through the set to find the best match from the kernel defined by the dissimilarity measure above. Results presented show better recognition results than conventional segmentation and template matching methods. ::: In the second topic, dynamic time alignment (DTA) based SVM gesture recognition is addressed. In particular, the proposed method combines DTA and SVM by establishing a new kernel. The gesture data is first projected into a common eigenspace formed by principal component analysis (PCA) and a distance measure is derived from the DTA. By incorporating DTA in the kernel function, general classification problems with variable-sized sequential data can be handled. ::: In the third topic, a C++ based gesture recognition application for the multi-touchpad is implemented. It uses the proposed gesture classification method along with a recursive neural networks approach to recognize definable gestures in real time, then runs an associated command. This application can further enable users with different disabilities or preferences to custom define gestures and enhance the functionality of the multi-touchpad. ::: Fourthly, an SVM-based classification method that uses the DTW to measure the similarity score is presented. The key contribution of this approach is the extension of trajectory based approaches to handle shape information, thereby enabling the expansion of the system's gesture vocabulary. It consists of two steps: converting a given set of frames into fixed-length vectors and training an SVM from the vectorized manifolds. Using shape information not only yields discrimination among various gestures, but also enables gestures that cannot be characterized solely based on their motion information to be classified, thus boosting overall recognition scores. ::: Finally, a computer vision based gesture command and communication system is developed. This system performs two major tasks: the first is to utilize the 3D traces of laser pointing devices as input to perform common keyboard and mouse control; the second is supplement free continuous gesture recognition, i.e., data gloves or other assistive devices are not necessary for 3D gestures recognition. As a result, the gesture can be used as a text entry system in wearable computers or mobile communication devices, though the recognition rate is lower thanthe approaches with the assistive tools. The purpose of this system is to develop new perceptual interfaces for human computer interaction based on visual input captured by computer vision systems, and to investigate how such interfaces can complement or replace traditional interfaces. ::: Original contributions of this work span the areas of SVMs and interpretation of computer sensory inputs, such as gestures for advanced HCI. In particular, we have addressed the following important issues: (1) ASM base kernels for shape recognition. (2) DTA based sequence kernels for gesture classification. (3) Recurrent neural networks (RNN). (4) Exploration of a customizable HCI. (5) Computer vision based 3D gesture recognition algorithms and system.
---
paper_title: Realtime online adaptive gesture recognition

paper_content:

We introduce an online adaptive algorithm for learning gesture models. By learning gesture models in an online fashion, the gesture recognition process is made more robust, and the need to train on a large training ensemble is obviated. Hidden Markov models are used to represent the spatial and temporal structure of the gesture. The usual output probability distributions-typically representing appearance-are trained at runtime exploiting the temporal structure (Markov model) that is either trained off-line or is explicitly hand-coded. In the early stages of runtime adaptation, contextural information derived from the application is used to bias the expectation as to which Markov state the system is in at any given time. We describe the Watch and Learn system, a computer vision system which is able to learn simple gestures online for interactive control.
---
paper_title: Gesture Recognition: A Survey

paper_content:

Gesture recognition pertains to recognizing meaningful expressions of motion by a human, involving the hands, arms, face, head, and/or body. It is of utmost importance in designing an intelligent and efficient human-computer interface. The applications of gesture recognition are manifold, ranging from sign language through medical rehabilitation to virtual reality. In this paper, we provide a survey on gesture recognition with particular emphasis on hand gestures and facial expressions. Applications involving hidden Markov models, particle filtering and condensation, finite-state machines, optical flow, skin color, and connectionist models are discussed in detail. Existing challenges and future research possibilities are also highlighted
---
paper_title: Visual recognition of aircraft marshalling signals using gesture phase analysis

paper_content:

Visual gesture recognition is one of the main areas of research in human-computer and human-robot interfaces. In this paper we present a novel visual gesture recognition method for aircraft marshalling signals. To capture hand motion information, we used a color-based tracking algorithm with an adaptive window for each frame. A feature selection algorithm was used to classify the motion information into four different gesture phases. By using the gesture phase information, we built the gesture model, which consisted of a symbol sequence and a number of probabilities. Each gesture model was learned from the longest common subsequence (LCS) of a set of symbol sequences and the probability of the symbols. A similarity measure using the proposed gesture model is presented that combines the deterministic and probabilistic matching scores. In the experiments we show the efficiency and accuracy of the proposed method.
---
paper_title: Modeling gesture salience as a hidden variable for coreference resolution and keyframe extraction

paper_content:

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and noncommunicative hand motions may confuse natural language processing (NLP) and impede learning. People have little difficulty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: Gesture patterns during speech repairs

paper_content:

Speech and gesture are two primary modes used in natural human communication; hence, they are important inputs for a multimodal interface to process. One of the challenges for multimodal interfaces is to accurately recognize the words in spontaneous speech. This is partly due to the presence of speech repairs, which seriously degrade the accuracy of current speech recognition systems. Based on the assumption that speech and gesture arise from the same thought process, we would expect to find patterns of gesture that co-occur with speech repairs that can be exploited by a multimodal processing system to more effectively process spontaneous speech. To evaluate this hypothesis, we have conducted a measurement study of gesture and speech repair data extracted from videotapes of natural dialogs. Although we have found that gestures do not always co-occur with speech repairs, we observed that modification gesture patterns have a high correlation with content replacement speech repairs, but rarely occur with content repetitions. These results suggest that gesture patterns can help us to classify different types of speech repairs in order to correct them more accurately.
---
paper_title: Segmentation and Recognition of Continuous Gestures

paper_content:

A novel method is introduced to segment and recognize time-varying human gestures from continuous video streams. Motion is represented by a 3D spatio-temporal surface based upon the evolution of a contour over time. The warping paths between the input signal and a set of gesture models are obtained using continuous dynamic programming and the boundary of a gesture is located by analyzing all possible gesture candidates during a specific period of time. Correlation and mutual information are employed to select the best candidate when more than one gesture is recognized at the same time period. The system has been implemented and tested on continuous gesture sequences containing 8 different gestures performed by 4 subjects. The results demonstrate that the proposed method is very effective, achieving a recognition rate of 95.9%.
---
paper_title: Discourse topic and gestural form

paper_content:

Coverbal gesture provides a channel for the visual expression of ideas. While some gestural emblems have culturally predefined forms (e.g., "thumbs up"), the relationship between gesture and meaning is, in general, not conventionalized. It is natural to ask whether such gestures can be interpreted in a speaker-independent way, or whether gestural form is determined by the speaker's idiosyncratic view of the discourse topic. We address this question using an audiovisual dataset across multiple speakers and topics. Our analysis employs a hierarchical Bayesian author-topic model, in which gestural patterns are stochastically generated by a mixture of speaker-specific and topic-specific priors. These gestural patterns are characterized using automatically-extracted visual features, based on spatio-temporal interest points. This framework detects significant cross-speaker patterns in gesture that are governed by the discourse topic, suggesting that even unstructured gesticulation can be interpreted across speakers. In addition, the success of this approach shows that the semantic characteristics of gesture can be detected via a low-level, interest point representation.
---
paper_title: Automated Posture Segmentation in Continuous Finger Spelling Recognition

paper_content:

Recognizing continuous finger spelling plays an important role in understanding sign language. There are two major phases in recognizing continuous finger spelling, which are posture segmentation and posture recognition. In the former, a continuous gesture sequence is decomposed into segments, which are then used for the latter to indentify corresponding characters. Among all the segments, beside valid postures corresponding to characters, there are also many movement epentheses, which appear between pairs of postures to move the hands from the end of one posture to the beginning of the next. In this paper, we propose a framework to split a continuous movement sequence into segments as well as to identify valid postures and movement epentheses. By using the velocity and signing rate based filter, we can obtain very good result with both high recall and precision rate.
---
paper_title: Multimodal model integration for sentence unit detection

paper_content:

In this paper, we adopt a direct modeling approach to utilize conversational gesture cues in detecting sentence boundaries, called SUs, in video taped conversations. We treat the detection of SUs as a classification task such that for each inter-word boundary, the classifier decides whether there is an SU boundary or not. In addition to gesture cues, we also utilize prosody and lexical knowledge sources. In this first investigation, we find that gesture features complement the prosodic and lexical knowledge sources for this task. By using all of the knowledge sources, the model is able to achieve the lowest overall SU detection error rate.
---
paper_title: BoVDW: Bag-of-Visual-and-Depth-Words for gesture recognition

paper_content:

We present a Bag-of-Visual-and-Depth-Words (BoVDW) model for gesture recognition, an extension of the Bag-of-Visual-Words (BoVW) model, that benefits from the multimodal fusion of visual and depth features. State-of-the-art RGB and depth features, including a new proposed depth descriptor, are analysed and combined in a late fusion fashion. The method is integrated in a continuous gesture recognition pipeline, where Dynamic Time Warping (DTW) algorithm is used to perform prior segmentation of gestures. Results of the method in public data sets, within our gesture recognition pipeline, show better performance in comparison to a standard BoVW model.
---
paper_title: A Compositional Model for Gesture Definition

paper_content:

The description of a gesture requires temporal analysis of values generated by input sensors and does not fit well the observer pattern traditionally used by frameworks to handle user input. The current solution is to embed particular gesture-based interactions, such as pinch-to-zoom, into frameworks by notifying when a whole gesture is detected. This approach suffers from a lack of flexibility unless the programmer performs explicit temporal analysis of raw sensors data. This paper proposes a compositional, declarative meta-model for gestures definition based on Petri Nets. Basic traits are used as building blocks for defining gestures; each one notifies the change of a feature value. A complex gesture is defined by the composition of other sub-gestures using a set of operators. The user interface behaviour can be associated to the recognition of the whole gesture or to any other sub-component, addressing the problem of granularity for the notification events. The meta-model can be instantiated for different gesture recognition supports and its definition has been validated through a proof of concept library. Sample applications have been developed for supporting multitouch gestures on iOS and full body gestures with Microsoft Kinect.
---
paper_title: Multimodal human discourse: gesture and speech

paper_content:

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues.
---
paper_title: Hand motion gestural oscillations and multimodal discourse

paper_content:

To develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion oscillatory gesture detection in natural speech and conversation. First, the hand motion trajectory signals are extracted from video. Second, a wavelet analysis based approach is presented to process the signals. In this approach, wavelet ridges are extracted from the responses of wavelet analysis for the hand motion trajectory signals, which can be used to characterize frequency properties of the hand motion signals. The hand motion oscillatory gestures can be extracted from these frequency properties. Finally, we relate the hand motion oscillatory gestures to the phases of speech and multimodal discourse analysis.We demonstrate the efficacy of the system on a real discourse dataset in which a subject described her action plan to an interlocutor. We extracted the oscillatory gestures from the x, y and z motion traces of both hands. We further demonstrate the power of gestural oscillation detection as a key to unlock the structure of the underlying discourse.
---
paper_title: Continuous Gesture Recognition using a Sparse Bayesian Classifier

paper_content:

An approach to recognise and segment 9 elementary gestures from a video input is proposed and it can be applied to continuous sign recognition. An isolated gesture is recognised by first converting a portion of video into a motion gradient orientation image and then classifying it into one of the 9 gestures by a sparse Bayesian classifier. The portion of video used is decided by using a sampling technique based on CONDENSATION framework. By doing so, gestures can be segmented from the video in a probabilistic manner. Experiments show that the proposed method can achieve accuracy around 90% in both isolated and continuous gesture recognition without using special equipment such as glove devices and the system can run in real-time.
---
paper_title: The catchment feature model: A device for multimodal fusion and a bridge between signal and sense

paper_content:

The catchment feature model addresses two questions in the field of multimodal interaction: how we bridge video and audio processing with the realities of human multimodal communication, and how information from the different modes may be fused. We argue from a detailed literature review that gestural research has clustered around manipulative and semaphoric use of the hands, motivate the catchment feature model psycholinguistic research, and present the model. In contrast to "whole gesture" recognition, the catchment feature model applies a feature decomposition approach that facilitates cross-modal fusion at the level of discourse planning and conceptualization. We present our experimental framework for catchment feature-based research, cite three concrete examples of catchment features, and propose new directions of multimodal research based on the model.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Capturing Expressive and Indicative Qualities of Conducting Gesture: An Application of Temporal Expectancy Models

paper_content:

Many event sequences in everyday human movement exhibit temporal structure: for instance, footsteps in walking, the striking of balls in a tennis match, the movements of a dancer set to rhythmic music, and the gestures of an orchestra conductor. These events generate prior expectancies regarding the occurrence of future events. Moreover, these expectancies play a critical role in conveying expressive qualities and communicative intent through the movement; thus they are of considerable interest in musical control contexts. To this end, we introduce a novel Bayesian framework which we call the temporal expectancy modeland use it to develop an analysis tool for capturing expressiveand indicativequalities of the conducting gesture based on temporal expectancies. The temporal expectancy model is a general dynamic Bayesian network (DBN) that can be used to encode prior knowledge regarding temporal structure to improve event segmentation. The conducting analysis tool infers beat and tempo, which are indicative and articulation which is expressive, as well as temporal expectancies regarding beat (ictusand preparationinstances) from conducting gesture. Experimental results using our analysis framework reveal a very strong correlation in how significantly the preparation expectancy builds up for staccato vs legato articulation, which bolsters the case for temporal expectancy as cognitive model for event anticipation, and as a key factor in the communication of expressive qualities of conducting gesture. Our system operates on data obtained from a marker based motion capture system, but can be easily adapted for more affordable technologies like video camera arrays.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis

paper_content:

Gesture and speech are co-expressive and complementary channels of a single human language system. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. We investigate the role of oscillatory/cyclical hand motions in `carrying' this image content. We present our work on the extraction of hand motion oscillation frequencies of gestures that accompany speech. The key challenges are that such motions are characterized by non-stationary oscillations, and multiple frequencies may be simultaneously extant. Also, the duration of the oscillations may be extended over very few cycles. We apply the windowed Fourier transform and wavelet transform to detect and extract gesticulatory oscillations. We tested these against synthetic signals (stationary and non-stationary) and real data sequences of gesticulatory hand movements in natural discourse. Our results show that both filters functioned well for the synthetic signals. For the real data, the wavelet bandpass filter bank is better for detecting and extracting hand gesture oscillations. We relate the hand motion oscillatory gestures detected by wavelet analysis to speech in natural conversation and apply to multimodal language analysis. We demonstrate the ability of our algorithm to extract gesticulatory oscillations and show how oscillatory gestures reveal portions of the multimodal discourse structure.
---
paper_title: Gesture in Automatic Discourse Processing

paper_content:

Computers cannot fully understand spoken language without access to the wide range of modalities that accompany speech. This thesis addresses the particularly expressive modality of hand gesture, and focuses on building structured statistical models at the intersection of speech, vision, and meaning. ::: My approach is distinguished in two key respects. First, gestural patterns are leveraged to discover parallel structures in the meaning of the associated speech. This differs from prior work that attempted to interpret individual gestures directly, an approach that was prone to a lack of generality across speakers. Second, I present novel, structured statistical models for multimodal language processing, which enable learning about gesture in its linguistic context, rather than in the abstract. ::: These ideas find successful application in a variety of language processing tasks: resolving ambiguous noun phrases, segmenting speech into topics, and producing keyframe summaries of spoken language. In all three cases, the addition of gestural features—extracted automatically from video—yields significantly improved performance over a state-of-the-art text-only alternative. This marks the first demonstration that hand gesture improves automatic discourse processing. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)
---
paper_title: Hand gesture symmetric behavior detection and analysis in natural conversation

paper_content:

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.
---
paper_title: Improved dynamic gesture segmentation for Thai sign language translation

paper_content:

Sign languages normally make use of both static and dynamic gestures to achieve communication goal. In automated continuous gesture recognition, one of the problems found is how to separate a transitional gesture from a meaningful dynamic gesture. In addition, a variation in speed, absolute position and orientation of a dynamic gesture makes an automatic recognition task nontrivial. This paper proposes a segmentation method that alleviates these problems. The method is based on the fact that many (meaningful) dynamic gestures as appeared in Thai and other sign languages, are performed in an approximately (quasi) periodic manner. Therefore, such dynamic gestures can be distinguished from a transitional gesture by means of Fourier analysis, performed on samples (discrete signal) of hand shapes, positions and orientation. The analysis, however, requires a good choice of data sample windowing. This paper describes a preprocessing method for estimation of the dynamic gesture data windowing period. The proposed method helps improving the accuracy of segmenting hand shape and location data samples into static, periodic and nonperiodic gestures. In addition, accuracy of the feature extracted from the Fourier analysis of a dynamic gesture signal is improved, thus increasing the recognition rate. Experimental results performed on a real data are included.
---
paper_title: Slideshow: Gesture-aware PPT presentation

paper_content:

Traditional keyboard and mouse based presentation prevents lecturers from interacting with the audiences freely and closely. In this paper, we propose a gesture-aware presentation tool named SlideShow to liberate lecturers from physical space constraints and make human-computer interaction more natural and convenient. In our system, gesture data is obtained by a handle controller with 3-axis accelerometer and gyro and transmitted to host-side through bluetooth, then we use Bayesian change point detection to segment continuous gesture series and HMM to recognize the gesture. In consequence Slideshow could carry out the corresponding operations on PowerPoint(PPT) to make a presentation, and operation states can be switched automatically and intelligently during the presentation. Both the experimental and testing results show our approach is practical, useful and convenient.
---
paper_title: Image-based gesture recognition with support vector machines

paper_content:

Recent advances in various display and virtual technologies, coupled with an explosion in available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities, among which gesture recognition is undoubtedly the most grammatically structured and complex. However, despite the abundance of novel interaction devices, the naturalness and efficiency of HCI has remained low. This is due in particular to the lack of robust sensory data interpretation techniques. To address the task of gesture recognition, this dissertation establishes novel probabilistic approaches based on support vector machines (SVM). Of special concern in this dissertation are the shapes of contact images on a multi-touch input device for both 2D and 3D. Five main topics are covered in this work. ::: The first topic deals with the hand pose recognition problem. To perform classification of different gestures, a recognition system must attempt to leverage between class variations (semantically varying gestures), while accommodating potentially large within-class variations (different hand poses to perform certain gestures). For recognition of gestures, a sequence of hand shapes should be recognized. We present a novel shape recognition approach using Active Shape Model (ASM) based matching and SVM based classification. Firstly, a set of correspondences between the reference shape and query image are identified through ASM. Next, a dissimilarity measure is created to measure how well any correspondence in the set aligns the reference shape and candidate shape in the query image. Finally, SVM classification is employed to search through the set to find the best match from the kernel defined by the dissimilarity measure above. Results presented show better recognition results than conventional segmentation and template matching methods. ::: In the second topic, dynamic time alignment (DTA) based SVM gesture recognition is addressed. In particular, the proposed method combines DTA and SVM by establishing a new kernel. The gesture data is first projected into a common eigenspace formed by principal component analysis (PCA) and a distance measure is derived from the DTA. By incorporating DTA in the kernel function, general classification problems with variable-sized sequential data can be handled. ::: In the third topic, a C++ based gesture recognition application for the multi-touchpad is implemented. It uses the proposed gesture classification method along with a recursive neural networks approach to recognize definable gestures in real time, then runs an associated command. This application can further enable users with different disabilities or preferences to custom define gestures and enhance the functionality of the multi-touchpad. ::: Fourthly, an SVM-based classification method that uses the DTW to measure the similarity score is presented. The key contribution of this approach is the extension of trajectory based approaches to handle shape information, thereby enabling the expansion of the system's gesture vocabulary. It consists of two steps: converting a given set of frames into fixed-length vectors and training an SVM from the vectorized manifolds. Using shape information not only yields discrimination among various gestures, but also enables gestures that cannot be characterized solely based on their motion information to be classified, thus boosting overall recognition scores. ::: Finally, a computer vision based gesture command and communication system is developed. This system performs two major tasks: the first is to utilize the 3D traces of laser pointing devices as input to perform common keyboard and mouse control; the second is supplement free continuous gesture recognition, i.e., data gloves or other assistive devices are not necessary for 3D gestures recognition. As a result, the gesture can be used as a text entry system in wearable computers or mobile communication devices, though the recognition rate is lower thanthe approaches with the assistive tools. The purpose of this system is to develop new perceptual interfaces for human computer interaction based on visual input captured by computer vision systems, and to investigate how such interfaces can complement or replace traditional interfaces. ::: Original contributions of this work span the areas of SVMs and interpretation of computer sensory inputs, such as gestures for advanced HCI. In particular, we have addressed the following important issues: (1) ASM base kernels for shape recognition. (2) DTA based sequence kernels for gesture classification. (3) Recurrent neural networks (RNN). (4) Exploration of a customizable HCI. (5) Computer vision based 3D gesture recognition algorithms and system.
---
paper_title: Realtime online adaptive gesture recognition

paper_content:

We introduce an online adaptive algorithm for learning gesture models. By learning gesture models in an online fashion, the gesture recognition process is made more robust, and the need to train on a large training ensemble is obviated. Hidden Markov models are used to represent the spatial and temporal structure of the gesture. The usual output probability distributions-typically representing appearance-are trained at runtime exploiting the temporal structure (Markov model) that is either trained off-line or is explicitly hand-coded. In the early stages of runtime adaptation, contextural information derived from the application is used to bias the expectation as to which Markov state the system is in at any given time. We describe the Watch and Learn system, a computer vision system which is able to learn simple gestures online for interactive control.
---
paper_title: Visual recognition of aircraft marshalling signals using gesture phase analysis

paper_content:

Visual gesture recognition is one of the main areas of research in human-computer and human-robot interfaces. In this paper we present a novel visual gesture recognition method for aircraft marshalling signals. To capture hand motion information, we used a color-based tracking algorithm with an adaptive window for each frame. A feature selection algorithm was used to classify the motion information into four different gesture phases. By using the gesture phase information, we built the gesture model, which consisted of a symbol sequence and a number of probabilities. Each gesture model was learned from the longest common subsequence (LCS) of a set of symbol sequences and the probability of the symbols. A similarity measure using the proposed gesture model is presented that combines the deterministic and probabilistic matching scores. In the experiments we show the efficiency and accuracy of the proposed method.
---
paper_title: Modeling gesture salience as a hidden variable for coreference resolution and keyframe extraction

paper_content:

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and noncommunicative hand motions may confuse natural language processing (NLP) and impede learning. People have little difficulty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: Gesture patterns during speech repairs

paper_content:

Speech and gesture are two primary modes used in natural human communication; hence, they are important inputs for a multimodal interface to process. One of the challenges for multimodal interfaces is to accurately recognize the words in spontaneous speech. This is partly due to the presence of speech repairs, which seriously degrade the accuracy of current speech recognition systems. Based on the assumption that speech and gesture arise from the same thought process, we would expect to find patterns of gesture that co-occur with speech repairs that can be exploited by a multimodal processing system to more effectively process spontaneous speech. To evaluate this hypothesis, we have conducted a measurement study of gesture and speech repair data extracted from videotapes of natural dialogs. Although we have found that gestures do not always co-occur with speech repairs, we observed that modification gesture patterns have a high correlation with content replacement speech repairs, but rarely occur with content repetitions. These results suggest that gesture patterns can help us to classify different types of speech repairs in order to correct them more accurately.
---
paper_title: Discourse topic and gestural form

paper_content:

Coverbal gesture provides a channel for the visual expression of ideas. While some gestural emblems have culturally predefined forms (e.g., "thumbs up"), the relationship between gesture and meaning is, in general, not conventionalized. It is natural to ask whether such gestures can be interpreted in a speaker-independent way, or whether gestural form is determined by the speaker's idiosyncratic view of the discourse topic. We address this question using an audiovisual dataset across multiple speakers and topics. Our analysis employs a hierarchical Bayesian author-topic model, in which gestural patterns are stochastically generated by a mixture of speaker-specific and topic-specific priors. These gestural patterns are characterized using automatically-extracted visual features, based on spatio-temporal interest points. This framework detects significant cross-speaker patterns in gesture that are governed by the discourse topic, suggesting that even unstructured gesticulation can be interpreted across speakers. In addition, the success of this approach shows that the semantic characteristics of gesture can be detected via a low-level, interest point representation.
---
paper_title: A Compositional Model for Gesture Definition

paper_content:

The description of a gesture requires temporal analysis of values generated by input sensors and does not fit well the observer pattern traditionally used by frameworks to handle user input. The current solution is to embed particular gesture-based interactions, such as pinch-to-zoom, into frameworks by notifying when a whole gesture is detected. This approach suffers from a lack of flexibility unless the programmer performs explicit temporal analysis of raw sensors data. This paper proposes a compositional, declarative meta-model for gestures definition based on Petri Nets. Basic traits are used as building blocks for defining gestures; each one notifies the change of a feature value. A complex gesture is defined by the composition of other sub-gestures using a set of operators. The user interface behaviour can be associated to the recognition of the whole gesture or to any other sub-component, addressing the problem of granularity for the notification events. The meta-model can be instantiated for different gesture recognition supports and its definition has been validated through a proof of concept library. Sample applications have been developed for supporting multitouch gestures on iOS and full body gestures with Microsoft Kinect.
---
paper_title: Temporal series recognition using a new neural network structure T-CombNET

paper_content:

We present a new neural network structure dedicated to the temporal data series recognition, called T-CombNET (Temporal CombNET). The T-CombNET is a modified and improved version of the CombNET-II structure, which was designed to deal with very large vocabulary character recognition. We incorporated a time normalization stage and used recurrent neural networks to enable CombNET-II to do the temporal analysis. Then we applied this new model T-CombNET to the hand gesture recognition task. The preliminary results show that the proposed structure is very efficient, obtaining a recognition rate of 99.41% in visual based kana hand alphabet recognition experiments.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Image-based gesture recognition with support vector machines

paper_content:

Recent advances in various display and virtual technologies, coupled with an explosion in available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities, among which gesture recognition is undoubtedly the most grammatically structured and complex. However, despite the abundance of novel interaction devices, the naturalness and efficiency of HCI has remained low. This is due in particular to the lack of robust sensory data interpretation techniques. To address the task of gesture recognition, this dissertation establishes novel probabilistic approaches based on support vector machines (SVM). Of special concern in this dissertation are the shapes of contact images on a multi-touch input device for both 2D and 3D. Five main topics are covered in this work. ::: The first topic deals with the hand pose recognition problem. To perform classification of different gestures, a recognition system must attempt to leverage between class variations (semantically varying gestures), while accommodating potentially large within-class variations (different hand poses to perform certain gestures). For recognition of gestures, a sequence of hand shapes should be recognized. We present a novel shape recognition approach using Active Shape Model (ASM) based matching and SVM based classification. Firstly, a set of correspondences between the reference shape and query image are identified through ASM. Next, a dissimilarity measure is created to measure how well any correspondence in the set aligns the reference shape and candidate shape in the query image. Finally, SVM classification is employed to search through the set to find the best match from the kernel defined by the dissimilarity measure above. Results presented show better recognition results than conventional segmentation and template matching methods. ::: In the second topic, dynamic time alignment (DTA) based SVM gesture recognition is addressed. In particular, the proposed method combines DTA and SVM by establishing a new kernel. The gesture data is first projected into a common eigenspace formed by principal component analysis (PCA) and a distance measure is derived from the DTA. By incorporating DTA in the kernel function, general classification problems with variable-sized sequential data can be handled. ::: In the third topic, a C++ based gesture recognition application for the multi-touchpad is implemented. It uses the proposed gesture classification method along with a recursive neural networks approach to recognize definable gestures in real time, then runs an associated command. This application can further enable users with different disabilities or preferences to custom define gestures and enhance the functionality of the multi-touchpad. ::: Fourthly, an SVM-based classification method that uses the DTW to measure the similarity score is presented. The key contribution of this approach is the extension of trajectory based approaches to handle shape information, thereby enabling the expansion of the system's gesture vocabulary. It consists of two steps: converting a given set of frames into fixed-length vectors and training an SVM from the vectorized manifolds. Using shape information not only yields discrimination among various gestures, but also enables gestures that cannot be characterized solely based on their motion information to be classified, thus boosting overall recognition scores. ::: Finally, a computer vision based gesture command and communication system is developed. This system performs two major tasks: the first is to utilize the 3D traces of laser pointing devices as input to perform common keyboard and mouse control; the second is supplement free continuous gesture recognition, i.e., data gloves or other assistive devices are not necessary for 3D gestures recognition. As a result, the gesture can be used as a text entry system in wearable computers or mobile communication devices, though the recognition rate is lower thanthe approaches with the assistive tools. The purpose of this system is to develop new perceptual interfaces for human computer interaction based on visual input captured by computer vision systems, and to investigate how such interfaces can complement or replace traditional interfaces. ::: Original contributions of this work span the areas of SVMs and interpretation of computer sensory inputs, such as gestures for advanced HCI. In particular, we have addressed the following important issues: (1) ASM base kernels for shape recognition. (2) DTA based sequence kernels for gesture classification. (3) Recurrent neural networks (RNN). (4) Exploration of a customizable HCI. (5) Computer vision based 3D gesture recognition algorithms and system.
---
paper_title: Modeling gesture salience as a hidden variable for coreference resolution and keyframe extraction

paper_content:

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and noncommunicative hand motions may confuse natural language processing (NLP) and impede learning. People have little difficulty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature.
---
paper_title: Segmentation and Recognition of Continuous Gestures

paper_content:

A novel method is introduced to segment and recognize time-varying human gestures from continuous video streams. Motion is represented by a 3D spatio-temporal surface based upon the evolution of a contour over time. The warping paths between the input signal and a set of gesture models are obtained using continuous dynamic programming and the boundary of a gesture is located by analyzing all possible gesture candidates during a specific period of time. Correlation and mutual information are employed to select the best candidate when more than one gesture is recognized at the same time period. The system has been implemented and tested on continuous gesture sequences containing 8 different gestures performed by 4 subjects. The results demonstrate that the proposed method is very effective, achieving a recognition rate of 95.9%.
---
paper_title: Discourse topic and gestural form

paper_content:

Coverbal gesture provides a channel for the visual expression of ideas. While some gestural emblems have culturally predefined forms (e.g., "thumbs up"), the relationship between gesture and meaning is, in general, not conventionalized. It is natural to ask whether such gestures can be interpreted in a speaker-independent way, or whether gestural form is determined by the speaker's idiosyncratic view of the discourse topic. We address this question using an audiovisual dataset across multiple speakers and topics. Our analysis employs a hierarchical Bayesian author-topic model, in which gestural patterns are stochastically generated by a mixture of speaker-specific and topic-specific priors. These gestural patterns are characterized using automatically-extracted visual features, based on spatio-temporal interest points. This framework detects significant cross-speaker patterns in gesture that are governed by the discourse topic, suggesting that even unstructured gesticulation can be interpreted across speakers. In addition, the success of this approach shows that the semantic characteristics of gesture can be detected via a low-level, interest point representation.
---
paper_title: Multimodal model integration for sentence unit detection

paper_content:

In this paper, we adopt a direct modeling approach to utilize conversational gesture cues in detecting sentence boundaries, called SUs, in video taped conversations. We treat the detection of SUs as a classification task such that for each inter-word boundary, the classifier decides whether there is an SU boundary or not. In addition to gesture cues, we also utilize prosody and lexical knowledge sources. In this first investigation, we find that gesture features complement the prosodic and lexical knowledge sources for this task. By using all of the knowledge sources, the model is able to achieve the lowest overall SU detection error rate.
---
paper_title: BoVDW: Bag-of-Visual-and-Depth-Words for gesture recognition

paper_content:

We present a Bag-of-Visual-and-Depth-Words (BoVDW) model for gesture recognition, an extension of the Bag-of-Visual-Words (BoVW) model, that benefits from the multimodal fusion of visual and depth features. State-of-the-art RGB and depth features, including a new proposed depth descriptor, are analysed and combined in a late fusion fashion. The method is integrated in a continuous gesture recognition pipeline, where Dynamic Time Warping (DTW) algorithm is used to perform prior segmentation of gestures. Results of the method in public data sets, within our gesture recognition pipeline, show better performance in comparison to a standard BoVW model.
---
paper_title: A Compositional Model for Gesture Definition

paper_content:

The description of a gesture requires temporal analysis of values generated by input sensors and does not fit well the observer pattern traditionally used by frameworks to handle user input. The current solution is to embed particular gesture-based interactions, such as pinch-to-zoom, into frameworks by notifying when a whole gesture is detected. This approach suffers from a lack of flexibility unless the programmer performs explicit temporal analysis of raw sensors data. This paper proposes a compositional, declarative meta-model for gestures definition based on Petri Nets. Basic traits are used as building blocks for defining gestures; each one notifies the change of a feature value. A complex gesture is defined by the composition of other sub-gestures using a set of operators. The user interface behaviour can be associated to the recognition of the whole gesture or to any other sub-component, addressing the problem of granularity for the notification events. The meta-model can be instantiated for different gesture recognition supports and its definition has been validated through a proof of concept library. Sample applications have been developed for supporting multitouch gestures on iOS and full body gestures with Microsoft Kinect.
---
paper_title: Continuous Gesture Recognition using a Sparse Bayesian Classifier

paper_content:

An approach to recognise and segment 9 elementary gestures from a video input is proposed and it can be applied to continuous sign recognition. An isolated gesture is recognised by first converting a portion of video into a motion gradient orientation image and then classifying it into one of the 9 gestures by a sparse Bayesian classifier. The portion of video used is decided by using a sampling technique based on CONDENSATION framework. By doing so, gestures can be segmented from the video in a probabilistic manner. Experiments show that the proposed method can achieve accuracy around 90% in both isolated and continuous gesture recognition without using special equipment such as glove devices and the system can run in real-time.
---
paper_title: Temporal series recognition using a new neural network structure T-CombNET

paper_content:

We present a new neural network structure dedicated to the temporal data series recognition, called T-CombNET (Temporal CombNET). The T-CombNET is a modified and improved version of the CombNET-II structure, which was designed to deal with very large vocabulary character recognition. We incorporated a time normalization stage and used recurrent neural networks to enable CombNET-II to do the temporal analysis. Then we applied this new model T-CombNET to the hand gesture recognition task. The preliminary results show that the proposed structure is very efficient, obtaining a recognition rate of 99.41% in visual based kana hand alphabet recognition experiments.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Capturing Expressive and Indicative Qualities of Conducting Gesture: An Application of Temporal Expectancy Models

paper_content:

Many event sequences in everyday human movement exhibit temporal structure: for instance, footsteps in walking, the striking of balls in a tennis match, the movements of a dancer set to rhythmic music, and the gestures of an orchestra conductor. These events generate prior expectancies regarding the occurrence of future events. Moreover, these expectancies play a critical role in conveying expressive qualities and communicative intent through the movement; thus they are of considerable interest in musical control contexts. To this end, we introduce a novel Bayesian framework which we call the temporal expectancy modeland use it to develop an analysis tool for capturing expressiveand indicativequalities of the conducting gesture based on temporal expectancies. The temporal expectancy model is a general dynamic Bayesian network (DBN) that can be used to encode prior knowledge regarding temporal structure to improve event segmentation. The conducting analysis tool infers beat and tempo, which are indicative and articulation which is expressive, as well as temporal expectancies regarding beat (ictusand preparationinstances) from conducting gesture. Experimental results using our analysis framework reveal a very strong correlation in how significantly the preparation expectancy builds up for staccato vs legato articulation, which bolsters the case for temporal expectancy as cognitive model for event anticipation, and as a key factor in the communication of expressive qualities of conducting gesture. Our system operates on data obtained from a marker based motion capture system, but can be easily adapted for more affordable technologies like video camera arrays.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Gesture in Automatic Discourse Processing

paper_content:

Computers cannot fully understand spoken language without access to the wide range of modalities that accompany speech. This thesis addresses the particularly expressive modality of hand gesture, and focuses on building structured statistical models at the intersection of speech, vision, and meaning. ::: My approach is distinguished in two key respects. First, gestural patterns are leveraged to discover parallel structures in the meaning of the associated speech. This differs from prior work that attempted to interpret individual gestures directly, an approach that was prone to a lack of generality across speakers. Second, I present novel, structured statistical models for multimodal language processing, which enable learning about gesture in its linguistic context, rather than in the abstract. ::: These ideas find successful application in a variety of language processing tasks: resolving ambiguous noun phrases, segmenting speech into topics, and producing keyframe summaries of spoken language. In all three cases, the addition of gestural features—extracted automatically from video—yields significantly improved performance over a state-of-the-art text-only alternative. This marks the first demonstration that hand gesture improves automatic discourse processing. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)
---
paper_title: Slideshow: Gesture-aware PPT presentation

paper_content:

Traditional keyboard and mouse based presentation prevents lecturers from interacting with the audiences freely and closely. In this paper, we propose a gesture-aware presentation tool named SlideShow to liberate lecturers from physical space constraints and make human-computer interaction more natural and convenient. In our system, gesture data is obtained by a handle controller with 3-axis accelerometer and gyro and transmitted to host-side through bluetooth, then we use Bayesian change point detection to segment continuous gesture series and HMM to recognize the gesture. In consequence Slideshow could carry out the corresponding operations on PowerPoint(PPT) to make a presentation, and operation states can be switched automatically and intelligently during the presentation. Both the experimental and testing results show our approach is practical, useful and convenient.
---
paper_title: Image-based gesture recognition with support vector machines

paper_content:

Recent advances in various display and virtual technologies, coupled with an explosion in available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities, among which gesture recognition is undoubtedly the most grammatically structured and complex. However, despite the abundance of novel interaction devices, the naturalness and efficiency of HCI has remained low. This is due in particular to the lack of robust sensory data interpretation techniques. To address the task of gesture recognition, this dissertation establishes novel probabilistic approaches based on support vector machines (SVM). Of special concern in this dissertation are the shapes of contact images on a multi-touch input device for both 2D and 3D. Five main topics are covered in this work. ::: The first topic deals with the hand pose recognition problem. To perform classification of different gestures, a recognition system must attempt to leverage between class variations (semantically varying gestures), while accommodating potentially large within-class variations (different hand poses to perform certain gestures). For recognition of gestures, a sequence of hand shapes should be recognized. We present a novel shape recognition approach using Active Shape Model (ASM) based matching and SVM based classification. Firstly, a set of correspondences between the reference shape and query image are identified through ASM. Next, a dissimilarity measure is created to measure how well any correspondence in the set aligns the reference shape and candidate shape in the query image. Finally, SVM classification is employed to search through the set to find the best match from the kernel defined by the dissimilarity measure above. Results presented show better recognition results than conventional segmentation and template matching methods. ::: In the second topic, dynamic time alignment (DTA) based SVM gesture recognition is addressed. In particular, the proposed method combines DTA and SVM by establishing a new kernel. The gesture data is first projected into a common eigenspace formed by principal component analysis (PCA) and a distance measure is derived from the DTA. By incorporating DTA in the kernel function, general classification problems with variable-sized sequential data can be handled. ::: In the third topic, a C++ based gesture recognition application for the multi-touchpad is implemented. It uses the proposed gesture classification method along with a recursive neural networks approach to recognize definable gestures in real time, then runs an associated command. This application can further enable users with different disabilities or preferences to custom define gestures and enhance the functionality of the multi-touchpad. ::: Fourthly, an SVM-based classification method that uses the DTW to measure the similarity score is presented. The key contribution of this approach is the extension of trajectory based approaches to handle shape information, thereby enabling the expansion of the system's gesture vocabulary. It consists of two steps: converting a given set of frames into fixed-length vectors and training an SVM from the vectorized manifolds. Using shape information not only yields discrimination among various gestures, but also enables gestures that cannot be characterized solely based on their motion information to be classified, thus boosting overall recognition scores. ::: Finally, a computer vision based gesture command and communication system is developed. This system performs two major tasks: the first is to utilize the 3D traces of laser pointing devices as input to perform common keyboard and mouse control; the second is supplement free continuous gesture recognition, i.e., data gloves or other assistive devices are not necessary for 3D gestures recognition. As a result, the gesture can be used as a text entry system in wearable computers or mobile communication devices, though the recognition rate is lower thanthe approaches with the assistive tools. The purpose of this system is to develop new perceptual interfaces for human computer interaction based on visual input captured by computer vision systems, and to investigate how such interfaces can complement or replace traditional interfaces. ::: Original contributions of this work span the areas of SVMs and interpretation of computer sensory inputs, such as gestures for advanced HCI. In particular, we have addressed the following important issues: (1) ASM base kernels for shape recognition. (2) DTA based sequence kernels for gesture classification. (3) Recurrent neural networks (RNN). (4) Exploration of a customizable HCI. (5) Computer vision based 3D gesture recognition algorithms and system.
---
paper_title: Realtime online adaptive gesture recognition

paper_content:

We introduce an online adaptive algorithm for learning gesture models. By learning gesture models in an online fashion, the gesture recognition process is made more robust, and the need to train on a large training ensemble is obviated. Hidden Markov models are used to represent the spatial and temporal structure of the gesture. The usual output probability distributions-typically representing appearance-are trained at runtime exploiting the temporal structure (Markov model) that is either trained off-line or is explicitly hand-coded. In the early stages of runtime adaptation, contextural information derived from the application is used to bias the expectation as to which Markov state the system is in at any given time. We describe the Watch and Learn system, a computer vision system which is able to learn simple gestures online for interactive control.
---
paper_title: Visual recognition of aircraft marshalling signals using gesture phase analysis

paper_content:

Visual gesture recognition is one of the main areas of research in human-computer and human-robot interfaces. In this paper we present a novel visual gesture recognition method for aircraft marshalling signals. To capture hand motion information, we used a color-based tracking algorithm with an adaptive window for each frame. A feature selection algorithm was used to classify the motion information into four different gesture phases. By using the gesture phase information, we built the gesture model, which consisted of a symbol sequence and a number of probabilities. Each gesture model was learned from the longest common subsequence (LCS) of a set of symbol sequences and the probability of the symbols. A similarity measure using the proposed gesture model is presented that combines the deterministic and probabilistic matching scores. In the experiments we show the efficiency and accuracy of the proposed method.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: Segmentation and Recognition of Continuous Gestures

paper_content:

A novel method is introduced to segment and recognize time-varying human gestures from continuous video streams. Motion is represented by a 3D spatio-temporal surface based upon the evolution of a contour over time. The warping paths between the input signal and a set of gesture models are obtained using continuous dynamic programming and the boundary of a gesture is located by analyzing all possible gesture candidates during a specific period of time. Correlation and mutual information are employed to select the best candidate when more than one gesture is recognized at the same time period. The system has been implemented and tested on continuous gesture sequences containing 8 different gestures performed by 4 subjects. The results demonstrate that the proposed method is very effective, achieving a recognition rate of 95.9%.
---
paper_title: Discourse topic and gestural form

paper_content:

Coverbal gesture provides a channel for the visual expression of ideas. While some gestural emblems have culturally predefined forms (e.g., "thumbs up"), the relationship between gesture and meaning is, in general, not conventionalized. It is natural to ask whether such gestures can be interpreted in a speaker-independent way, or whether gestural form is determined by the speaker's idiosyncratic view of the discourse topic. We address this question using an audiovisual dataset across multiple speakers and topics. Our analysis employs a hierarchical Bayesian author-topic model, in which gestural patterns are stochastically generated by a mixture of speaker-specific and topic-specific priors. These gestural patterns are characterized using automatically-extracted visual features, based on spatio-temporal interest points. This framework detects significant cross-speaker patterns in gesture that are governed by the discourse topic, suggesting that even unstructured gesticulation can be interpreted across speakers. In addition, the success of this approach shows that the semantic characteristics of gesture can be detected via a low-level, interest point representation.
---
paper_title: Automated Posture Segmentation in Continuous Finger Spelling Recognition

paper_content:

Recognizing continuous finger spelling plays an important role in understanding sign language. There are two major phases in recognizing continuous finger spelling, which are posture segmentation and posture recognition. In the former, a continuous gesture sequence is decomposed into segments, which are then used for the latter to indentify corresponding characters. Among all the segments, beside valid postures corresponding to characters, there are also many movement epentheses, which appear between pairs of postures to move the hands from the end of one posture to the beginning of the next. In this paper, we propose a framework to split a continuous movement sequence into segments as well as to identify valid postures and movement epentheses. By using the velocity and signing rate based filter, we can obtain very good result with both high recall and precision rate.
---
paper_title: Multimodal model integration for sentence unit detection

paper_content:

In this paper, we adopt a direct modeling approach to utilize conversational gesture cues in detecting sentence boundaries, called SUs, in video taped conversations. We treat the detection of SUs as a classification task such that for each inter-word boundary, the classifier decides whether there is an SU boundary or not. In addition to gesture cues, we also utilize prosody and lexical knowledge sources. In this first investigation, we find that gesture features complement the prosodic and lexical knowledge sources for this task. By using all of the knowledge sources, the model is able to achieve the lowest overall SU detection error rate.
---
paper_title: BoVDW: Bag-of-Visual-and-Depth-Words for gesture recognition

paper_content:

We present a Bag-of-Visual-and-Depth-Words (BoVDW) model for gesture recognition, an extension of the Bag-of-Visual-Words (BoVW) model, that benefits from the multimodal fusion of visual and depth features. State-of-the-art RGB and depth features, including a new proposed depth descriptor, are analysed and combined in a late fusion fashion. The method is integrated in a continuous gesture recognition pipeline, where Dynamic Time Warping (DTW) algorithm is used to perform prior segmentation of gestures. Results of the method in public data sets, within our gesture recognition pipeline, show better performance in comparison to a standard BoVW model.
---
paper_title: Hand motion gestural oscillations and multimodal discourse

paper_content:

To develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion oscillatory gesture detection in natural speech and conversation. First, the hand motion trajectory signals are extracted from video. Second, a wavelet analysis based approach is presented to process the signals. In this approach, wavelet ridges are extracted from the responses of wavelet analysis for the hand motion trajectory signals, which can be used to characterize frequency properties of the hand motion signals. The hand motion oscillatory gestures can be extracted from these frequency properties. Finally, we relate the hand motion oscillatory gestures to the phases of speech and multimodal discourse analysis.We demonstrate the efficacy of the system on a real discourse dataset in which a subject described her action plan to an interlocutor. We extracted the oscillatory gestures from the x, y and z motion traces of both hands. We further demonstrate the power of gestural oscillation detection as a key to unlock the structure of the underlying discourse.
---
paper_title: Continuous Gesture Recognition using a Sparse Bayesian Classifier

paper_content:

An approach to recognise and segment 9 elementary gestures from a video input is proposed and it can be applied to continuous sign recognition. An isolated gesture is recognised by first converting a portion of video into a motion gradient orientation image and then classifying it into one of the 9 gestures by a sparse Bayesian classifier. The portion of video used is decided by using a sampling technique based on CONDENSATION framework. By doing so, gestures can be segmented from the video in a probabilistic manner. Experiments show that the proposed method can achieve accuracy around 90% in both isolated and continuous gesture recognition without using special equipment such as glove devices and the system can run in real-time.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: Temporal series recognition using a new neural network structure T-CombNET

paper_content:

We present a new neural network structure dedicated to the temporal data series recognition, called T-CombNET (Temporal CombNET). The T-CombNET is a modified and improved version of the CombNET-II structure, which was designed to deal with very large vocabulary character recognition. We incorporated a time normalization stage and used recurrent neural networks to enable CombNET-II to do the temporal analysis. Then we applied this new model T-CombNET to the hand gesture recognition task. The preliminary results show that the proposed structure is very efficient, obtaining a recognition rate of 99.41% in visual based kana hand alphabet recognition experiments.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Capturing Expressive and Indicative Qualities of Conducting Gesture: An Application of Temporal Expectancy Models

paper_content:

Many event sequences in everyday human movement exhibit temporal structure: for instance, footsteps in walking, the striking of balls in a tennis match, the movements of a dancer set to rhythmic music, and the gestures of an orchestra conductor. These events generate prior expectancies regarding the occurrence of future events. Moreover, these expectancies play a critical role in conveying expressive qualities and communicative intent through the movement; thus they are of considerable interest in musical control contexts. To this end, we introduce a novel Bayesian framework which we call the temporal expectancy modeland use it to develop an analysis tool for capturing expressiveand indicativequalities of the conducting gesture based on temporal expectancies. The temporal expectancy model is a general dynamic Bayesian network (DBN) that can be used to encode prior knowledge regarding temporal structure to improve event segmentation. The conducting analysis tool infers beat and tempo, which are indicative and articulation which is expressive, as well as temporal expectancies regarding beat (ictusand preparationinstances) from conducting gesture. Experimental results using our analysis framework reveal a very strong correlation in how significantly the preparation expectancy builds up for staccato vs legato articulation, which bolsters the case for temporal expectancy as cognitive model for event anticipation, and as a key factor in the communication of expressive qualities of conducting gesture. Our system operates on data obtained from a marker based motion capture system, but can be easily adapted for more affordable technologies like video camera arrays.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis

paper_content:

Gesture and speech are co-expressive and complementary channels of a single human language system. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. We investigate the role of oscillatory/cyclical hand motions in `carrying' this image content. We present our work on the extraction of hand motion oscillation frequencies of gestures that accompany speech. The key challenges are that such motions are characterized by non-stationary oscillations, and multiple frequencies may be simultaneously extant. Also, the duration of the oscillations may be extended over very few cycles. We apply the windowed Fourier transform and wavelet transform to detect and extract gesticulatory oscillations. We tested these against synthetic signals (stationary and non-stationary) and real data sequences of gesticulatory hand movements in natural discourse. Our results show that both filters functioned well for the synthetic signals. For the real data, the wavelet bandpass filter bank is better for detecting and extracting hand gesture oscillations. We relate the hand motion oscillatory gestures detected by wavelet analysis to speech in natural conversation and apply to multimodal language analysis. We demonstrate the ability of our algorithm to extract gesticulatory oscillations and show how oscillatory gestures reveal portions of the multimodal discourse structure.
---
paper_title: Gesture in Automatic Discourse Processing

paper_content:

Computers cannot fully understand spoken language without access to the wide range of modalities that accompany speech. This thesis addresses the particularly expressive modality of hand gesture, and focuses on building structured statistical models at the intersection of speech, vision, and meaning. ::: My approach is distinguished in two key respects. First, gestural patterns are leveraged to discover parallel structures in the meaning of the associated speech. This differs from prior work that attempted to interpret individual gestures directly, an approach that was prone to a lack of generality across speakers. Second, I present novel, structured statistical models for multimodal language processing, which enable learning about gesture in its linguistic context, rather than in the abstract. ::: These ideas find successful application in a variety of language processing tasks: resolving ambiguous noun phrases, segmenting speech into topics, and producing keyframe summaries of spoken language. In all three cases, the addition of gestural features—extracted automatically from video—yields significantly improved performance over a state-of-the-art text-only alternative. This marks the first demonstration that hand gesture improves automatic discourse processing. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)
---
paper_title: Hand gesture symmetric behavior detection and analysis in natural conversation

paper_content:

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.
---
paper_title: Improved dynamic gesture segmentation for Thai sign language translation

paper_content:

Sign languages normally make use of both static and dynamic gestures to achieve communication goal. In automated continuous gesture recognition, one of the problems found is how to separate a transitional gesture from a meaningful dynamic gesture. In addition, a variation in speed, absolute position and orientation of a dynamic gesture makes an automatic recognition task nontrivial. This paper proposes a segmentation method that alleviates these problems. The method is based on the fact that many (meaningful) dynamic gestures as appeared in Thai and other sign languages, are performed in an approximately (quasi) periodic manner. Therefore, such dynamic gestures can be distinguished from a transitional gesture by means of Fourier analysis, performed on samples (discrete signal) of hand shapes, positions and orientation. The analysis, however, requires a good choice of data sample windowing. This paper describes a preprocessing method for estimation of the dynamic gesture data windowing period. The proposed method helps improving the accuracy of segmenting hand shape and location data samples into static, periodic and nonperiodic gestures. In addition, accuracy of the feature extracted from the Fourier analysis of a dynamic gesture signal is improved, thus increasing the recognition rate. Experimental results performed on a real data are included.
---
paper_title: Slideshow: Gesture-aware PPT presentation

paper_content:

Traditional keyboard and mouse based presentation prevents lecturers from interacting with the audiences freely and closely. In this paper, we propose a gesture-aware presentation tool named SlideShow to liberate lecturers from physical space constraints and make human-computer interaction more natural and convenient. In our system, gesture data is obtained by a handle controller with 3-axis accelerometer and gyro and transmitted to host-side through bluetooth, then we use Bayesian change point detection to segment continuous gesture series and HMM to recognize the gesture. In consequence Slideshow could carry out the corresponding operations on PowerPoint(PPT) to make a presentation, and operation states can be switched automatically and intelligently during the presentation. Both the experimental and testing results show our approach is practical, useful and convenient.
---
paper_title: Image-based gesture recognition with support vector machines

paper_content:

Recent advances in various display and virtual technologies, coupled with an explosion in available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities, among which gesture recognition is undoubtedly the most grammatically structured and complex. However, despite the abundance of novel interaction devices, the naturalness and efficiency of HCI has remained low. This is due in particular to the lack of robust sensory data interpretation techniques. To address the task of gesture recognition, this dissertation establishes novel probabilistic approaches based on support vector machines (SVM). Of special concern in this dissertation are the shapes of contact images on a multi-touch input device for both 2D and 3D. Five main topics are covered in this work. ::: The first topic deals with the hand pose recognition problem. To perform classification of different gestures, a recognition system must attempt to leverage between class variations (semantically varying gestures), while accommodating potentially large within-class variations (different hand poses to perform certain gestures). For recognition of gestures, a sequence of hand shapes should be recognized. We present a novel shape recognition approach using Active Shape Model (ASM) based matching and SVM based classification. Firstly, a set of correspondences between the reference shape and query image are identified through ASM. Next, a dissimilarity measure is created to measure how well any correspondence in the set aligns the reference shape and candidate shape in the query image. Finally, SVM classification is employed to search through the set to find the best match from the kernel defined by the dissimilarity measure above. Results presented show better recognition results than conventional segmentation and template matching methods. ::: In the second topic, dynamic time alignment (DTA) based SVM gesture recognition is addressed. In particular, the proposed method combines DTA and SVM by establishing a new kernel. The gesture data is first projected into a common eigenspace formed by principal component analysis (PCA) and a distance measure is derived from the DTA. By incorporating DTA in the kernel function, general classification problems with variable-sized sequential data can be handled. ::: In the third topic, a C++ based gesture recognition application for the multi-touchpad is implemented. It uses the proposed gesture classification method along with a recursive neural networks approach to recognize definable gestures in real time, then runs an associated command. This application can further enable users with different disabilities or preferences to custom define gestures and enhance the functionality of the multi-touchpad. ::: Fourthly, an SVM-based classification method that uses the DTW to measure the similarity score is presented. The key contribution of this approach is the extension of trajectory based approaches to handle shape information, thereby enabling the expansion of the system's gesture vocabulary. It consists of two steps: converting a given set of frames into fixed-length vectors and training an SVM from the vectorized manifolds. Using shape information not only yields discrimination among various gestures, but also enables gestures that cannot be characterized solely based on their motion information to be classified, thus boosting overall recognition scores. ::: Finally, a computer vision based gesture command and communication system is developed. This system performs two major tasks: the first is to utilize the 3D traces of laser pointing devices as input to perform common keyboard and mouse control; the second is supplement free continuous gesture recognition, i.e., data gloves or other assistive devices are not necessary for 3D gestures recognition. As a result, the gesture can be used as a text entry system in wearable computers or mobile communication devices, though the recognition rate is lower thanthe approaches with the assistive tools. The purpose of this system is to develop new perceptual interfaces for human computer interaction based on visual input captured by computer vision systems, and to investigate how such interfaces can complement or replace traditional interfaces. ::: Original contributions of this work span the areas of SVMs and interpretation of computer sensory inputs, such as gestures for advanced HCI. In particular, we have addressed the following important issues: (1) ASM base kernels for shape recognition. (2) DTA based sequence kernels for gesture classification. (3) Recurrent neural networks (RNN). (4) Exploration of a customizable HCI. (5) Computer vision based 3D gesture recognition algorithms and system.
---
paper_title: Realtime online adaptive gesture recognition

paper_content:

We introduce an online adaptive algorithm for learning gesture models. By learning gesture models in an online fashion, the gesture recognition process is made more robust, and the need to train on a large training ensemble is obviated. Hidden Markov models are used to represent the spatial and temporal structure of the gesture. The usual output probability distributions-typically representing appearance-are trained at runtime exploiting the temporal structure (Markov model) that is either trained off-line or is explicitly hand-coded. In the early stages of runtime adaptation, contextural information derived from the application is used to bias the expectation as to which Markov state the system is in at any given time. We describe the Watch and Learn system, a computer vision system which is able to learn simple gestures online for interactive control.
---
paper_title: Visual recognition of aircraft marshalling signals using gesture phase analysis

paper_content:

Visual gesture recognition is one of the main areas of research in human-computer and human-robot interfaces. In this paper we present a novel visual gesture recognition method for aircraft marshalling signals. To capture hand motion information, we used a color-based tracking algorithm with an adaptive window for each frame. A feature selection algorithm was used to classify the motion information into four different gesture phases. By using the gesture phase information, we built the gesture model, which consisted of a symbol sequence and a number of probabilities. Each gesture model was learned from the longest common subsequence (LCS) of a set of symbol sequences and the probability of the symbols. A similarity measure using the proposed gesture model is presented that combines the deterministic and probabilistic matching scores. In the experiments we show the efficiency and accuracy of the proposed method.
---
paper_title: Modeling gesture salience as a hidden variable for coreference resolution and keyframe extraction

paper_content:

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and noncommunicative hand motions may confuse natural language processing (NLP) and impede learning. People have little difficulty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Capturing Expressive and Indicative Qualities of Conducting Gesture: An Application of Temporal Expectancy Models

paper_content:

Many event sequences in everyday human movement exhibit temporal structure: for instance, footsteps in walking, the striking of balls in a tennis match, the movements of a dancer set to rhythmic music, and the gestures of an orchestra conductor. These events generate prior expectancies regarding the occurrence of future events. Moreover, these expectancies play a critical role in conveying expressive qualities and communicative intent through the movement; thus they are of considerable interest in musical control contexts. To this end, we introduce a novel Bayesian framework which we call the temporal expectancy modeland use it to develop an analysis tool for capturing expressiveand indicativequalities of the conducting gesture based on temporal expectancies. The temporal expectancy model is a general dynamic Bayesian network (DBN) that can be used to encode prior knowledge regarding temporal structure to improve event segmentation. The conducting analysis tool infers beat and tempo, which are indicative and articulation which is expressive, as well as temporal expectancies regarding beat (ictusand preparationinstances) from conducting gesture. Experimental results using our analysis framework reveal a very strong correlation in how significantly the preparation expectancy builds up for staccato vs legato articulation, which bolsters the case for temporal expectancy as cognitive model for event anticipation, and as a key factor in the communication of expressive qualities of conducting gesture. Our system operates on data obtained from a marker based motion capture system, but can be easily adapted for more affordable technologies like video camera arrays.
---
paper_title: Realtime online adaptive gesture recognition

paper_content:

We introduce an online adaptive algorithm for learning gesture models. By learning gesture models in an online fashion, the gesture recognition process is made more robust, and the need to train on a large training ensemble is obviated. Hidden Markov models are used to represent the spatial and temporal structure of the gesture. The usual output probability distributions-typically representing appearance-are trained at runtime exploiting the temporal structure (Markov model) that is either trained off-line or is explicitly hand-coded. In the early stages of runtime adaptation, contextural information derived from the application is used to bias the expectation as to which Markov state the system is in at any given time. We describe the Watch and Learn system, a computer vision system which is able to learn simple gestures online for interactive control.
---
paper_title: Visual recognition of aircraft marshalling signals using gesture phase analysis

paper_content:

Visual gesture recognition is one of the main areas of research in human-computer and human-robot interfaces. In this paper we present a novel visual gesture recognition method for aircraft marshalling signals. To capture hand motion information, we used a color-based tracking algorithm with an adaptive window for each frame. A feature selection algorithm was used to classify the motion information into four different gesture phases. By using the gesture phase information, we built the gesture model, which consisted of a symbol sequence and a number of probabilities. Each gesture model was learned from the longest common subsequence (LCS) of a set of symbol sequences and the probability of the symbols. A similarity measure using the proposed gesture model is presented that combines the deterministic and probabilistic matching scores. In the experiments we show the efficiency and accuracy of the proposed method.
---
paper_title: Automated Posture Segmentation in Continuous Finger Spelling Recognition

paper_content:

Recognizing continuous finger spelling plays an important role in understanding sign language. There are two major phases in recognizing continuous finger spelling, which are posture segmentation and posture recognition. In the former, a continuous gesture sequence is decomposed into segments, which are then used for the latter to indentify corresponding characters. Among all the segments, beside valid postures corresponding to characters, there are also many movement epentheses, which appear between pairs of postures to move the hands from the end of one posture to the beginning of the next. In this paper, we propose a framework to split a continuous movement sequence into segments as well as to identify valid postures and movement epentheses. By using the velocity and signing rate based filter, we can obtain very good result with both high recall and precision rate.
---
paper_title: Hand motion gestural oscillations and multimodal discourse

paper_content:

To develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion oscillatory gesture detection in natural speech and conversation. First, the hand motion trajectory signals are extracted from video. Second, a wavelet analysis based approach is presented to process the signals. In this approach, wavelet ridges are extracted from the responses of wavelet analysis for the hand motion trajectory signals, which can be used to characterize frequency properties of the hand motion signals. The hand motion oscillatory gestures can be extracted from these frequency properties. Finally, we relate the hand motion oscillatory gestures to the phases of speech and multimodal discourse analysis.We demonstrate the efficacy of the system on a real discourse dataset in which a subject described her action plan to an interlocutor. We extracted the oscillatory gestures from the x, y and z motion traces of both hands. We further demonstrate the power of gestural oscillation detection as a key to unlock the structure of the underlying discourse.
---
paper_title: Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis

paper_content:

Gesture and speech are co-expressive and complementary channels of a single human language system. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. We investigate the role of oscillatory/cyclical hand motions in `carrying' this image content. We present our work on the extraction of hand motion oscillation frequencies of gestures that accompany speech. The key challenges are that such motions are characterized by non-stationary oscillations, and multiple frequencies may be simultaneously extant. Also, the duration of the oscillations may be extended over very few cycles. We apply the windowed Fourier transform and wavelet transform to detect and extract gesticulatory oscillations. We tested these against synthetic signals (stationary and non-stationary) and real data sequences of gesticulatory hand movements in natural discourse. Our results show that both filters functioned well for the synthetic signals. For the real data, the wavelet bandpass filter bank is better for detecting and extracting hand gesture oscillations. We relate the hand motion oscillatory gestures detected by wavelet analysis to speech in natural conversation and apply to multimodal language analysis. We demonstrate the ability of our algorithm to extract gesticulatory oscillations and show how oscillatory gestures reveal portions of the multimodal discourse structure.
---
paper_title: Hand gesture symmetric behavior detection and analysis in natural conversation

paper_content:

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.
---
paper_title: Improved dynamic gesture segmentation for Thai sign language translation

paper_content:

Sign languages normally make use of both static and dynamic gestures to achieve communication goal. In automated continuous gesture recognition, one of the problems found is how to separate a transitional gesture from a meaningful dynamic gesture. In addition, a variation in speed, absolute position and orientation of a dynamic gesture makes an automatic recognition task nontrivial. This paper proposes a segmentation method that alleviates these problems. The method is based on the fact that many (meaningful) dynamic gestures as appeared in Thai and other sign languages, are performed in an approximately (quasi) periodic manner. Therefore, such dynamic gestures can be distinguished from a transitional gesture by means of Fourier analysis, performed on samples (discrete signal) of hand shapes, positions and orientation. The analysis, however, requires a good choice of data sample windowing. This paper describes a preprocessing method for estimation of the dynamic gesture data windowing period. The proposed method helps improving the accuracy of segmenting hand shape and location data samples into static, periodic and nonperiodic gestures. In addition, accuracy of the feature extracted from the Fourier analysis of a dynamic gesture signal is improved, thus increasing the recognition rate. Experimental results performed on a real data are included.
---
paper_title: Slideshow: Gesture-aware PPT presentation

paper_content:

Traditional keyboard and mouse based presentation prevents lecturers from interacting with the audiences freely and closely. In this paper, we propose a gesture-aware presentation tool named SlideShow to liberate lecturers from physical space constraints and make human-computer interaction more natural and convenient. In our system, gesture data is obtained by a handle controller with 3-axis accelerometer and gyro and transmitted to host-side through bluetooth, then we use Bayesian change point detection to segment continuous gesture series and HMM to recognize the gesture. In consequence Slideshow could carry out the corresponding operations on PowerPoint(PPT) to make a presentation, and operation states can be switched automatically and intelligently during the presentation. Both the experimental and testing results show our approach is practical, useful and convenient.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: Segmentation and Recognition of Continuous Gestures

paper_content:

A novel method is introduced to segment and recognize time-varying human gestures from continuous video streams. Motion is represented by a 3D spatio-temporal surface based upon the evolution of a contour over time. The warping paths between the input signal and a set of gesture models are obtained using continuous dynamic programming and the boundary of a gesture is located by analyzing all possible gesture candidates during a specific period of time. Correlation and mutual information are employed to select the best candidate when more than one gesture is recognized at the same time period. The system has been implemented and tested on continuous gesture sequences containing 8 different gestures performed by 4 subjects. The results demonstrate that the proposed method is very effective, achieving a recognition rate of 95.9%.
---
paper_title: Automated Posture Segmentation in Continuous Finger Spelling Recognition

paper_content:

Recognizing continuous finger spelling plays an important role in understanding sign language. There are two major phases in recognizing continuous finger spelling, which are posture segmentation and posture recognition. In the former, a continuous gesture sequence is decomposed into segments, which are then used for the latter to indentify corresponding characters. Among all the segments, beside valid postures corresponding to characters, there are also many movement epentheses, which appear between pairs of postures to move the hands from the end of one posture to the beginning of the next. In this paper, we propose a framework to split a continuous movement sequence into segments as well as to identify valid postures and movement epentheses. By using the velocity and signing rate based filter, we can obtain very good result with both high recall and precision rate.
---
paper_title: BoVDW: Bag-of-Visual-and-Depth-Words for gesture recognition

paper_content:

We present a Bag-of-Visual-and-Depth-Words (BoVDW) model for gesture recognition, an extension of the Bag-of-Visual-Words (BoVW) model, that benefits from the multimodal fusion of visual and depth features. State-of-the-art RGB and depth features, including a new proposed depth descriptor, are analysed and combined in a late fusion fashion. The method is integrated in a continuous gesture recognition pipeline, where Dynamic Time Warping (DTW) algorithm is used to perform prior segmentation of gestures. Results of the method in public data sets, within our gesture recognition pipeline, show better performance in comparison to a standard BoVW model.
---
paper_title: Hand motion gestural oscillations and multimodal discourse

paper_content:

To develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion oscillatory gesture detection in natural speech and conversation. First, the hand motion trajectory signals are extracted from video. Second, a wavelet analysis based approach is presented to process the signals. In this approach, wavelet ridges are extracted from the responses of wavelet analysis for the hand motion trajectory signals, which can be used to characterize frequency properties of the hand motion signals. The hand motion oscillatory gestures can be extracted from these frequency properties. Finally, we relate the hand motion oscillatory gestures to the phases of speech and multimodal discourse analysis.We demonstrate the efficacy of the system on a real discourse dataset in which a subject described her action plan to an interlocutor. We extracted the oscillatory gestures from the x, y and z motion traces of both hands. We further demonstrate the power of gestural oscillation detection as a key to unlock the structure of the underlying discourse.
---
paper_title: Continuous Gesture Recognition using a Sparse Bayesian Classifier

paper_content:

An approach to recognise and segment 9 elementary gestures from a video input is proposed and it can be applied to continuous sign recognition. An isolated gesture is recognised by first converting a portion of video into a motion gradient orientation image and then classifying it into one of the 9 gestures by a sparse Bayesian classifier. The portion of video used is decided by using a sampling technique based on CONDENSATION framework. By doing so, gestures can be segmented from the video in a probabilistic manner. Experiments show that the proposed method can achieve accuracy around 90% in both isolated and continuous gesture recognition without using special equipment such as glove devices and the system can run in real-time.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis

paper_content:

Gesture and speech are co-expressive and complementary channels of a single human language system. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. We investigate the role of oscillatory/cyclical hand motions in `carrying' this image content. We present our work on the extraction of hand motion oscillation frequencies of gestures that accompany speech. The key challenges are that such motions are characterized by non-stationary oscillations, and multiple frequencies may be simultaneously extant. Also, the duration of the oscillations may be extended over very few cycles. We apply the windowed Fourier transform and wavelet transform to detect and extract gesticulatory oscillations. We tested these against synthetic signals (stationary and non-stationary) and real data sequences of gesticulatory hand movements in natural discourse. Our results show that both filters functioned well for the synthetic signals. For the real data, the wavelet bandpass filter bank is better for detecting and extracting hand gesture oscillations. We relate the hand motion oscillatory gestures detected by wavelet analysis to speech in natural conversation and apply to multimodal language analysis. We demonstrate the ability of our algorithm to extract gesticulatory oscillations and show how oscillatory gestures reveal portions of the multimodal discourse structure.
---
paper_title: Hand gesture symmetric behavior detection and analysis in natural conversation

paper_content:

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.
---
paper_title: Improved dynamic gesture segmentation for Thai sign language translation

paper_content:

Sign languages normally make use of both static and dynamic gestures to achieve communication goal. In automated continuous gesture recognition, one of the problems found is how to separate a transitional gesture from a meaningful dynamic gesture. In addition, a variation in speed, absolute position and orientation of a dynamic gesture makes an automatic recognition task nontrivial. This paper proposes a segmentation method that alleviates these problems. The method is based on the fact that many (meaningful) dynamic gestures as appeared in Thai and other sign languages, are performed in an approximately (quasi) periodic manner. Therefore, such dynamic gestures can be distinguished from a transitional gesture by means of Fourier analysis, performed on samples (discrete signal) of hand shapes, positions and orientation. The analysis, however, requires a good choice of data sample windowing. This paper describes a preprocessing method for estimation of the dynamic gesture data windowing period. The proposed method helps improving the accuracy of segmenting hand shape and location data samples into static, periodic and nonperiodic gestures. In addition, accuracy of the feature extracted from the Fourier analysis of a dynamic gesture signal is improved, thus increasing the recognition rate. Experimental results performed on a real data are included.
---
paper_title: Slideshow: Gesture-aware PPT presentation

paper_content:

Traditional keyboard and mouse based presentation prevents lecturers from interacting with the audiences freely and closely. In this paper, we propose a gesture-aware presentation tool named SlideShow to liberate lecturers from physical space constraints and make human-computer interaction more natural and convenient. In our system, gesture data is obtained by a handle controller with 3-axis accelerometer and gyro and transmitted to host-side through bluetooth, then we use Bayesian change point detection to segment continuous gesture series and HMM to recognize the gesture. In consequence Slideshow could carry out the corresponding operations on PowerPoint(PPT) to make a presentation, and operation states can be switched automatically and intelligently during the presentation. Both the experimental and testing results show our approach is practical, useful and convenient.
---
paper_title: Segmentation and Recognition of Continuous Gestures

paper_content:

A novel method is introduced to segment and recognize time-varying human gestures from continuous video streams. Motion is represented by a 3D spatio-temporal surface based upon the evolution of a contour over time. The warping paths between the input signal and a set of gesture models are obtained using continuous dynamic programming and the boundary of a gesture is located by analyzing all possible gesture candidates during a specific period of time. Correlation and mutual information are employed to select the best candidate when more than one gesture is recognized at the same time period. The system has been implemented and tested on continuous gesture sequences containing 8 different gestures performed by 4 subjects. The results demonstrate that the proposed method is very effective, achieving a recognition rate of 95.9%.
---
paper_title: Automated Posture Segmentation in Continuous Finger Spelling Recognition

paper_content:

Recognizing continuous finger spelling plays an important role in understanding sign language. There are two major phases in recognizing continuous finger spelling, which are posture segmentation and posture recognition. In the former, a continuous gesture sequence is decomposed into segments, which are then used for the latter to indentify corresponding characters. Among all the segments, beside valid postures corresponding to characters, there are also many movement epentheses, which appear between pairs of postures to move the hands from the end of one posture to the beginning of the next. In this paper, we propose a framework to split a continuous movement sequence into segments as well as to identify valid postures and movement epentheses. By using the velocity and signing rate based filter, we can obtain very good result with both high recall and precision rate.
---
paper_title: Multimodal model integration for sentence unit detection

paper_content:

In this paper, we adopt a direct modeling approach to utilize conversational gesture cues in detecting sentence boundaries, called SUs, in video taped conversations. We treat the detection of SUs as a classification task such that for each inter-word boundary, the classifier decides whether there is an SU boundary or not. In addition to gesture cues, we also utilize prosody and lexical knowledge sources. In this first investigation, we find that gesture features complement the prosodic and lexical knowledge sources for this task. By using all of the knowledge sources, the model is able to achieve the lowest overall SU detection error rate.
---
paper_title: BoVDW: Bag-of-Visual-and-Depth-Words for gesture recognition

paper_content:

We present a Bag-of-Visual-and-Depth-Words (BoVDW) model for gesture recognition, an extension of the Bag-of-Visual-Words (BoVW) model, that benefits from the multimodal fusion of visual and depth features. State-of-the-art RGB and depth features, including a new proposed depth descriptor, are analysed and combined in a late fusion fashion. The method is integrated in a continuous gesture recognition pipeline, where Dynamic Time Warping (DTW) algorithm is used to perform prior segmentation of gestures. Results of the method in public data sets, within our gesture recognition pipeline, show better performance in comparison to a standard BoVW model.
---
paper_title: A Compositional Model for Gesture Definition

paper_content:

The description of a gesture requires temporal analysis of values generated by input sensors and does not fit well the observer pattern traditionally used by frameworks to handle user input. The current solution is to embed particular gesture-based interactions, such as pinch-to-zoom, into frameworks by notifying when a whole gesture is detected. This approach suffers from a lack of flexibility unless the programmer performs explicit temporal analysis of raw sensors data. This paper proposes a compositional, declarative meta-model for gestures definition based on Petri Nets. Basic traits are used as building blocks for defining gestures; each one notifies the change of a feature value. A complex gesture is defined by the composition of other sub-gestures using a set of operators. The user interface behaviour can be associated to the recognition of the whole gesture or to any other sub-component, addressing the problem of granularity for the notification events. The meta-model can be instantiated for different gesture recognition supports and its definition has been validated through a proof of concept library. Sample applications have been developed for supporting multitouch gestures on iOS and full body gestures with Microsoft Kinect.
---
paper_title: Multimodal human discourse: gesture and speech

paper_content:

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the gesture research done to date, and present our work on the cross-modal cues for discourse segmentation in free-form gesticulation accompanying speech in natural conversation as a new paradigm for such multimodal interaction. The basis for this integration is the psycholinguistic concept of the coequal generation of gesture and speech from the same semantic intent. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. We perform two independent sets of analyses on the video and audio data: video and audio analysis to extract segmentation cues, and expert transcription of the speech and gesture data by microanalyzing the videotape using a frame-accurate videoplayer to correlate the speech with the gestural entities. We compare the results of both analyses to identify the cues accessible in the gestural and audio data that correlate well with the expert psycholinguistic analysis. We show that "handedness" and the kind of symmetry in two-handed gestures provide effective supersegmental discourse cues.
---
paper_title: Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders

paper_content:

The previous literature has suggested that the hand movement in co-speech gestures and signs consists of a series of phases with qualitatively different dynamic characteristics. In this paper, we propose a syntagmatic rule system for movement phases that applies to both co-speech gestures and signs. Descriptive criteria for the rule system were developed for the analysis video-recorded continuous production of signs and gesture. It involves segmenting a stream of body movement into phases and identifying different phase types. Two human coders used the criteria to analyze signs and cospeech gestures that are produced in natural discourse. It was found that the criteria yielded good inter-coder reliability. These criteria can be used for the technology of automatic recognition of signs and co-speech gestures in order to segment continuous production and identify the potentially meaningbearing phase.
---
paper_title: Hand motion gestural oscillations and multimodal discourse

paper_content:

To develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion oscillatory gesture detection in natural speech and conversation. First, the hand motion trajectory signals are extracted from video. Second, a wavelet analysis based approach is presented to process the signals. In this approach, wavelet ridges are extracted from the responses of wavelet analysis for the hand motion trajectory signals, which can be used to characterize frequency properties of the hand motion signals. The hand motion oscillatory gestures can be extracted from these frequency properties. Finally, we relate the hand motion oscillatory gestures to the phases of speech and multimodal discourse analysis.We demonstrate the efficacy of the system on a real discourse dataset in which a subject described her action plan to an interlocutor. We extracted the oscillatory gestures from the x, y and z motion traces of both hands. We further demonstrate the power of gestural oscillation detection as a key to unlock the structure of the underlying discourse.
---
paper_title: Continuous Gesture Recognition using a Sparse Bayesian Classifier

paper_content:

An approach to recognise and segment 9 elementary gestures from a video input is proposed and it can be applied to continuous sign recognition. An isolated gesture is recognised by first converting a portion of video into a motion gradient orientation image and then classifying it into one of the 9 gestures by a sparse Bayesian classifier. The portion of video used is decided by using a sampling technique based on CONDENSATION framework. By doing so, gestures can be segmented from the video in a probabilistic manner. Experiments show that the proposed method can achieve accuracy around 90% in both isolated and continuous gesture recognition without using special equipment such as glove devices and the system can run in real-time.
---
paper_title: The catchment feature model: A device for multimodal fusion and a bridge between signal and sense

paper_content:

The catchment feature model addresses two questions in the field of multimodal interaction: how we bridge video and audio processing with the realities of human multimodal communication, and how information from the different modes may be fused. We argue from a detailed literature review that gestural research has clustered around manipulative and semaphoric use of the hands, motivate the catchment feature model psycholinguistic research, and present the model. In contrast to "whole gesture" recognition, the catchment feature model applies a feature decomposition approach that facilitates cross-modal fusion at the level of discourse planning and conceptualization. We present our experimental framework for catchment feature-based research, cite three concrete examples of catchment features, and propose new directions of multimodal research based on the model.
---
paper_title: Simultaneous Gesture Segmentation and Recognition based on Forward Spotting Accumulative HMMs

paper_content:

In this paper, we propose a forward spotting scheme that executes gesture segmentation and recognition simultaneously by detecting start point. By using competitive differential observation probability, sliding window and accumulative HMMs, we apply the proposed method to recognize the upper-body gestures for controlling the curtains and lights in a smart home environment.
---
paper_title: Temporal series recognition using a new neural network structure T-CombNET

paper_content:

We present a new neural network structure dedicated to the temporal data series recognition, called T-CombNET (Temporal CombNET). The T-CombNET is a modified and improved version of the CombNET-II structure, which was designed to deal with very large vocabulary character recognition. We incorporated a time normalization stage and used recurrent neural networks to enable CombNET-II to do the temporal analysis. Then we applied this new model T-CombNET to the hand gesture recognition task. The preliminary results show that the proposed structure is very efficient, obtaining a recognition rate of 99.41% in visual based kana hand alphabet recognition experiments.
---
paper_title: Recovering the temporal structure of natural gesture

paper_content:

A method for the recovery of the temporal structure and phases in natural gesture is presented. The work is motivated by recent developments in the theory of natural gesture which have identified several key aspects of gesture important to communication. In particular, gesticulation during conversation can be coarsely characterized as periods of bi-phasic or tri-phasic gesture separated by a rest state. We first present an automatic procedure for hypothesizing plausible rest state configurations of a speaker; the method uses the repetition of subsequences to indicate potential rest states. Second, we develop a state-based parsing algorithm used to both select among candidate rest stares and to parse an incoming video stream into bi-phasic and multi-phasic gestures. We present results from examples of story-telling speakers.
---
paper_title: Capturing Expressive and Indicative Qualities of Conducting Gesture: An Application of Temporal Expectancy Models

paper_content:

Many event sequences in everyday human movement exhibit temporal structure: for instance, footsteps in walking, the striking of balls in a tennis match, the movements of a dancer set to rhythmic music, and the gestures of an orchestra conductor. These events generate prior expectancies regarding the occurrence of future events. Moreover, these expectancies play a critical role in conveying expressive qualities and communicative intent through the movement; thus they are of considerable interest in musical control contexts. To this end, we introduce a novel Bayesian framework which we call the temporal expectancy modeland use it to develop an analysis tool for capturing expressiveand indicativequalities of the conducting gesture based on temporal expectancies. The temporal expectancy model is a general dynamic Bayesian network (DBN) that can be used to encode prior knowledge regarding temporal structure to improve event segmentation. The conducting analysis tool infers beat and tempo, which are indicative and articulation which is expressive, as well as temporal expectancies regarding beat (ictusand preparationinstances) from conducting gesture. Experimental results using our analysis framework reveal a very strong correlation in how significantly the preparation expectancy builds up for staccato vs legato articulation, which bolsters the case for temporal expectancy as cognitive model for event anticipation, and as a key factor in the communication of expressive qualities of conducting gesture. Our system operates on data obtained from a marker based motion capture system, but can be easily adapted for more affordable technologies like video camera arrays.
---
paper_title: Exploiting prosodic structuring of coverbal gesticulation

paper_content:

Although gesture recognition has been studied extensively, communicative, affective, and biometrical "utility" of natural gesticulation remains relatively unexplored. One of the main reasons for that is the modeling complexity of spontaneous gestures. While lexical information in speech provides additional cues for disambiguating gestures, it does not cover rich paralinguistic domain. This paper offers initial findings from a large corpus of natural monologues about prosodic structuring between frequent beat-like strokes and concurrent speech. Using a set of audio-visual features in an HMM-based formulation, we are able to improve the discrimination between visually similar gestures. Those types of articulatory strokes represent different communicative functions. The analysis is based on the temporal alignment of detected vocal perturbations and the concurrent hand movement. As a supplementary result, we show that recognized articulatory strokes may be used for quantifying gesturing behavior.
---
paper_title: Hand Motion Gesture Frequency Properties and Multimodal Discourse Analysis

paper_content:

Gesture and speech are co-expressive and complementary channels of a single human language system. While speech carries the major load of symbolic presentation, gesture provides the imagistic content. We investigate the role of oscillatory/cyclical hand motions in `carrying' this image content. We present our work on the extraction of hand motion oscillation frequencies of gestures that accompany speech. The key challenges are that such motions are characterized by non-stationary oscillations, and multiple frequencies may be simultaneously extant. Also, the duration of the oscillations may be extended over very few cycles. We apply the windowed Fourier transform and wavelet transform to detect and extract gesticulatory oscillations. We tested these against synthetic signals (stationary and non-stationary) and real data sequences of gesticulatory hand movements in natural discourse. Our results show that both filters functioned well for the synthetic signals. For the real data, the wavelet bandpass filter bank is better for detecting and extracting hand gesture oscillations. We relate the hand motion oscillatory gestures detected by wavelet analysis to speech in natural conversation and apply to multimodal language analysis. We demonstrate the ability of our algorithm to extract gesticulatory oscillations and show how oscillatory gestures reveal portions of the multimodal discourse structure.
---
paper_title: Gesture in Automatic Discourse Processing

paper_content:

Computers cannot fully understand spoken language without access to the wide range of modalities that accompany speech. This thesis addresses the particularly expressive modality of hand gesture, and focuses on building structured statistical models at the intersection of speech, vision, and meaning. ::: My approach is distinguished in two key respects. First, gestural patterns are leveraged to discover parallel structures in the meaning of the associated speech. This differs from prior work that attempted to interpret individual gestures directly, an approach that was prone to a lack of generality across speakers. Second, I present novel, structured statistical models for multimodal language processing, which enable learning about gesture in its linguistic context, rather than in the abstract. ::: These ideas find successful application in a variety of language processing tasks: resolving ambiguous noun phrases, segmenting speech into topics, and producing keyframe summaries of spoken language. In all three cases, the addition of gestural features—extracted automatically from video—yields significantly improved performance over a state-of-the-art text-only alternative. This marks the first demonstration that hand gesture improves automatic discourse processing. (Copies available exclusively from MIT Libraries, Rm. 14-0551, Cambridge, MA 02139-4307. Ph. 617-253-5668; Fax 617-253-1690.)
---
paper_title: Hand gesture symmetric behavior detection and analysis in natural conversation

paper_content:

We present an experimental investigation into the phenomenon of gestural symmetry for two-handed gestures accompanying speech. We describe an approach to compute hand motion symmetries based on the correlation computations. Local symmetries are detected using a windowing operation. We demonstrate that the selection of a smaller window size results in better sensitivity to local symmetries at the expense of noise in the form of spurious symmetries and 'symmetry dropoffs'. Our algorithm applies a 'hole filling' post process to address these detection problems. We examine the role of the detected motion symmetries of two-handed gestures in the structuring of speech. We compared discourse segments corresponding to extracted symmetries in two natural conversations against a discourse analysis by expert psycholinguistic coders. These comparisons illustrate the effectiveness of the symmetry feature for the understanding of underlying discourse structure. We believe that this basic characteristic of two-handed gestures accompanying speech must be incorporated in any multimodal interaction system involving two-handed gestures and speech.
---
paper_title: Improved dynamic gesture segmentation for Thai sign language translation

paper_content:

Sign languages normally make use of both static and dynamic gestures to achieve communication goal. In automated continuous gesture recognition, one of the problems found is how to separate a transitional gesture from a meaningful dynamic gesture. In addition, a variation in speed, absolute position and orientation of a dynamic gesture makes an automatic recognition task nontrivial. This paper proposes a segmentation method that alleviates these problems. The method is based on the fact that many (meaningful) dynamic gestures as appeared in Thai and other sign languages, are performed in an approximately (quasi) periodic manner. Therefore, such dynamic gestures can be distinguished from a transitional gesture by means of Fourier analysis, performed on samples (discrete signal) of hand shapes, positions and orientation. The analysis, however, requires a good choice of data sample windowing. This paper describes a preprocessing method for estimation of the dynamic gesture data windowing period. The proposed method helps improving the accuracy of segmenting hand shape and location data samples into static, periodic and nonperiodic gestures. In addition, accuracy of the feature extracted from the Fourier analysis of a dynamic gesture signal is improved, thus increasing the recognition rate. Experimental results performed on a real data are included.
---
paper_title: Slideshow: Gesture-aware PPT presentation

paper_content:

Traditional keyboard and mouse based presentation prevents lecturers from interacting with the audiences freely and closely. In this paper, we propose a gesture-aware presentation tool named SlideShow to liberate lecturers from physical space constraints and make human-computer interaction more natural and convenient. In our system, gesture data is obtained by a handle controller with 3-axis accelerometer and gyro and transmitted to host-side through bluetooth, then we use Bayesian change point detection to segment continuous gesture series and HMM to recognize the gesture. In consequence Slideshow could carry out the corresponding operations on PowerPoint(PPT) to make a presentation, and operation states can be switched automatically and intelligently during the presentation. Both the experimental and testing results show our approach is practical, useful and convenient.
---
paper_title: Visual recognition of aircraft marshalling signals using gesture phase analysis

paper_content:

Visual gesture recognition is one of the main areas of research in human-computer and human-robot interfaces. In this paper we present a novel visual gesture recognition method for aircraft marshalling signals. To capture hand motion information, we used a color-based tracking algorithm with an adaptive window for each frame. A feature selection algorithm was used to classify the motion information into four different gesture phases. By using the gesture phase information, we built the gesture model, which consisted of a symbol sequence and a number of probabilities. Each gesture model was learned from the longest common subsequence (LCS) of a set of symbol sequences and the probability of the symbols. A similarity measure using the proposed gesture model is presented that combines the deterministic and probabilistic matching scores. In the experiments we show the efficiency and accuracy of the proposed method.
---
paper_title: Modeling gesture salience as a hidden variable for coreference resolution and keyframe extraction

paper_content:

Gesture is a non-verbal modality that can contribute crucial information to the understanding of natural language. But not all gestures are informative, and noncommunicative hand motions may confuse natural language processing (NLP) and impede learning. People have little difficulty ignoring irrelevant hand movements and focusing on meaningful gestures, suggesting that an automatic system could also be trained to perform this task. However, the informativeness of a gesture is context-dependent and labeling enough data to cover all cases would be expensive. We present conditional modality fusion, a conditional hidden-variable model that learns to predict which gestures are salient for coreference resolution, the task of determining whether two noun phrases refer to the same semantic entity. Moreover, our approach uses only coreference annotations, and not annotations of gesture salience itself. We show that gesture features improve performance on coreference resolution, and that by attending only to gestures that are salient, our method achieves further significant gains. In addition, we show that the model of gesture salience learned in the context of coreference accords with human intuition, by demonstrating that gestures judged to be salient by our model can be used successfully to create multimedia keyframe summaries of video. These summaries are similar to those created by human raters, and significantly outperform summaries produced by baselines from the literature.
---
paper_title: Prosody based audiovisual coanalysis for coverbal gesture recognition

paper_content:

Despite recent advances in vision-based gesture recognition, its applications remain largely limited to artificially defined and well-articulated gesture signs used for human-computer interaction. A key reason for this is the low recognition rates for "natural" gesticulation. Previous attempts of using speech cues to reduce error-proneness of visual classification have been mostly limited to keyword-gesture coanalysis. Such scheme inherits complexity and delays associated with natural language processing. This paper offers a novel "signal-level" perspective, where prosodic manifestations in speech and hand kinematics are considered as a basis for coanalyzing loosely coupled modalities. We present a computational framework for improving continuous gesture recognition based on two phenomena that capture voluntary (coarticulation) and involuntary (physiological) contributions of prosodic synchronization. Physiological constraints, manifested as signal interruptions during multimodal production, are exploited in an audiovisual feature integration framework using hidden Markov models. Coarticulation is analyzed using a Bayesian network of naive classifiers to explore alignment of intonationally prominent speech segments and hand kinematics. The efficacy of the proposed approach was demonstrated on a multimodal corpus created from the Weather Channel broadcast. Both schemas were found to contribute uniquely by reducing different error types, which subsequently improves the performance of continuous gesture recognition.
---
paper_title: Gesture patterns during speech repairs

paper_content:

Speech and gesture are two primary modes used in natural human communication; hence, they are important inputs for a multimodal interface to process. One of the challenges for multimodal interfaces is to accurately recognize the words in spontaneous speech. This is partly due to the presence of speech repairs, which seriously degrade the accuracy of current speech recognition systems. Based on the assumption that speech and gesture arise from the same thought process, we would expect to find patterns of gesture that co-occur with speech repairs that can be exploited by a multimodal processing system to more effectively process spontaneous speech. To evaluate this hypothesis, we have conducted a measurement study of gesture and speech repair data extracted from videotapes of natural dialogs. Although we have found that gestures do not always co-occur with speech repairs, we observed that modification gesture patterns have a high correlation with content replacement speech repairs, but rarely occur with content repetitions. These results suggest that gesture patterns can help us to classify different types of speech repairs in order to correct them more accurately.
---

Title: A Review of Temporal Aspects of Hand Gesture Analysis Applied to Discourse Analysis and Natural Conversation

Section 1: INTRODUCTION
Description 1: Discuss the evolution of computer science in relation to hand gesture analysis and its applications, explain the need for new HCI methods, and introduce the scope and organization of the paper.

Section 2: FUNDAMENTAL CONCEPTS
Description 2: Present fundamental concepts of automated gesture analysis, psycholinguistic analysis, and the theory of gestures from linguistics.

Section 3: Automated Gesture Analysis
Description 3: Define key concepts such as gesture, posture, and movement, provide classifications of gesture elements, and discuss differences between linguistic and automation perspectives.

Section 4: Psycholinguistic Analysis
Description 4: Explain psycholinguistics, the necessary consideration of gestures in language study, and the typical process for psycholinguistic analysis.

Section 5: Theory of Gestures
Description 5: Discuss the theoretical framework of gesture phases and the types of gestures as categorized by McNeill.

Section 6: Methodology: Systematic Literature Review
Description 6: Describe the process of systematic literature review, including planning, conducting the review, and reporting phases.

Section 7: Planning the review
Description 7: Outline the planning phase, detailing the review protocol, research questions, search strategies, and inclusion/exclusion criteria.

Section 8: Conducting the review
Description 8: Present the results of the conducted review, listing the number of studies selected and providing an initial analysis.

Section 9: Results and Discussion
Description 9: Analyze the temporal aspects of gesture analysis, the development of gesture analysis studies through the years, and the applications found in the review.

Section 10: Applications
Description 10: Discuss the major applications of gesture analysis, including the recognition of predefined gestures, gesticulation analysis, and perceptual analysis.

Section 11: Data Acquisition Method
Description 11: Describe methods for data acquisition in gesture analysis, comparing sensor-based and vision-based approaches.

Section 12: Feature Representation Method
Description 12: Explain various methods used to represent gesture features based on hand position and other derived features.

Section 13: Type of Analysis
Description 13: Classify the types of gesture analysis into recognition of previously segmented gestures, gesture segmentation and recognition, and other types of analyses.

Section 14: Methods for Result Analysis
Description 14: Detail strategies for evaluating the results of gesture analysis, focusing on recognition rates, error types, and comparison with human labeled videos.

Section 15: FINAL CONSIDERATIONS
Description 15: Summarize the contributions of the review, highlighting its focus on temporal aspects, different application types, and the systematic approach of various analysis strategies.