What is the Best Automated Metric for Text to Motion Generation?
Jordan Voas Yili Wang Qixing Huang Raymond Mooney
There is growing interest in generating skeleton-based human motions from
natural language descriptions. While most efforts have focused on developing
better neural architectures for this task, there has been no significant work
on determining the proper evaluation metric. Human evaluation is the ultimate
accuracy measure for this task, and automated metrics should correlate well
with human quality judgments. Since descriptions are compatible with many
motions, determining the right metric is critical for evaluating and designing
effective generative models. This paper systematically studies which metrics
best align with human evaluations and proposes new metrics that align even
better. Our findings indicate that none of the metrics currently used for this
task show even a moderate correlation with human judgments on a sample level.
However, for assessing average model performance, commonly used metrics such as
R-Precision and less-used coordinate errors show strong correlations.
Additionally, several recently developed metrics are not recommended due to
their low correlation compared to alternatives. We also introduce a novel
metric based on a multimodal BERT-like model, MoBERT, which offers strongly
human-correlated sample-level evaluations while maintaining near-perfect
model-level correlation. Our results demonstrate that this new metric exhibits
extensive benefits over all current alternatives.