The GENEA Challenge 2023: A large scale evaluation of gesture generation
  models in monadic and dyadic settings
Taras Kucherenko Rajmund Nagy Youngwoo Yoon Jieyeon Woo Teodor Nikolov Mihail Tsakov Gustav Eje Henter
This paper reports on the GENEA Challenge 2023, in which participating teams
built speech-driven gesture-generation systems using the same speech and motion
dataset, followed by a joint evaluation. This year's challenge provided data on
both sides of a dyadic interaction, allowing teams to generate full-body motion
for an agent given its speech (text and audio) and the speech and motion of the
interlocutor. We evaluated 12 submissions and 2 baselines together with
held-out motion-capture data in several large-scale user studies. The studies
focused on three aspects: 1) the human-likeness of the motion, 2) the
appropriateness of the motion for the agent's own speech whilst controlling for
the human-likeness of the motion, and 3) the appropriateness of the motion for
the behaviour of the interlocutor in the interaction, using a setup that
controls for both the human-likeness of the motion and the agent's own speech.
We found a large span in human-likeness between challenge submissions, with a
few systems rated close to human mocap. Appropriateness seems far from being
solved, with most submissions performing in a narrow range slightly above
chance, far behind natural motion. The effect of the interlocutor is even more
subtle, with submitted systems at best performing barely above chance.
Interestingly, a dyadic system being highly appropriate for agent speech does
not necessarily imply high appropriateness for the interlocutor. Additional
material is available via the project website at
https://svito-zar.github.io/GENEAchallenge2023/ .