Guided Attention for Interpretable Motion Captioning
Karim Radouane Andon Tchechmedjiev Sylvie Ranwez Julien Lagarde
While much effort has been invested in generating human motion from text,
relatively few studies have been dedicated to the reverse direction, that is,
generating text from motion. Much of the research focuses on maximizing
generation quality without any regard for the interpretability of the
architectures, particularly regarding the influence of particular body parts in
the generation and the temporal synchronization of words with specific
movements and actions. This study explores the combination of movement encoders
with spatio-temporal attention models and proposes strategies to guide the
attention during training to highlight perceptually pertinent areas of the
skeleton in time. We show that adding guided attention with adaptive gate leads
to interpretable captioning while improving performance compared to higher
parameter-count non-interpretable SOTA systems. On the KIT MLD dataset, we
obtain a BLEU@4 of 24.4% (SOTA+6%), a ROUGE-L of 58.30% (SOTA +14.1%), a CIDEr
of 112.10 (SOTA +32.6) and a Bertscore of 41.20% (SOTA +18.20%). On HumanML3D,
we obtain a BLEU@4 of 25.00 (SOTA +2.7%), a ROUGE-L score of 55.4% (SOTA
+6.1%), a CIDEr of 61.6 (SOTA -10.9%), a Bertscore of 40.3% (SOTA +2.5%). Our
code implementation and reproduction details will be soon available at
https://github.com/rd20karim/M2T-Interpretable/tree/main.