Co-Speech Gesture Detection through Multi-phase Sequence Labeling
Esam Ghaleb Ilya Burenko Marlou Rasenberg Wim Pouw Peter Uhrig Judith Holler Ivan Toni Aslı Özyürek Raquel Fernández
Gestures are integral components of face-to-face communication. They unfold
over time, often following predictable movement phases of preparation, stroke,
and retraction. Yet, the prevalent approach to automatic gesture detection
treats the problem as binary classification, classifying a segment as either
containing a gesture or not, thus failing to capture its inherently sequential
and contextual nature. To address this, we introduce a novel framework that
reframes the task as a multi-phase sequence labeling problem rather than binary
classification. Our model processes sequences of skeletal movements over time
windows, uses Transformer encoders to learn contextual embeddings, and
leverages Conditional Random Fields to perform sequence labeling. We evaluate
our proposal on a large dataset of diverse co-speech gestures in task-oriented
face-to-face dialogues. The results consistently demonstrate that our method
significantly outperforms strong baseline models in detecting gesture strokes.
Furthermore, applying Transformer encoders to learn contextual embeddings from
movement sequences substantially improves gesture unit detection. These results
highlight our framework's capacity to capture the fine-grained dynamics of
co-speech gesture phases, paving the way for more nuanced and accurate gesture
detection and analysis.