MPE4G: Multimodal Pretrained Encoder for Co-Speech Gesture Generation
Gwantae Kim Seonghyeok Noh Insung Ham Hanseok Ko
When virtual agents interact with humans, gestures are crucial to delivering
their intentions with speech. Previous multimodal co-speech gesture generation
models required encoded features of all modalities to generate gestures. If
some input modalities are removed or contain noise, the model may not generate
the gestures properly. To acquire robust and generalized encodings, we propose
a novel framework with a multimodal pre-trained encoder for co-speech gesture
generation. In the proposed method, the multi-head-attention-based encoder is
trained with self-supervised learning to contain the information on each
modality. Moreover, we collect full-body gestures that consist of 3D joint
rotations to improve visualization and apply gestures to the extensible body
model. Through the series of experiments and human evaluation, the proposed
method renders realistic co-speech gestures not only when all input modalities
are given but also when the input modalities are missing or noisy.