Interactive Conversational Head Generation
Mohan Zhou Yalong Bai Wei Zhang Ting Yao Tiejun Zhao
We introduce a new conversation head generation benchmark for synthesizing
behaviors of a single interlocutor in a face-to-face conversation. The
capability to automatically synthesize interlocutors which can participate in
long and multi-turn conversations is vital and offer benefits for various
applications, including digital humans, virtual agents, and social robots.
While existing research primarily focuses on talking head generation (one-way
interaction), hindering the ability to create a digital human for conversation
(two-way) interaction due to the absence of listening and interaction parts. In
this work, we construct two datasets to address this issue, ``ViCo'' for
independent talking and listening head generation tasks at the sentence level,
and ``ViCo-X'', for synthesizing interlocutors in multi-turn conversational
scenarios. Based on ViCo and ViCo-X, we define three novel tasks targeting the
interaction modeling during the face-to-face conversation: 1) responsive
listening head generation making listeners respond actively to the speaker with
non-verbal signals, 2) expressive talking head generation guiding speakers to
be aware of listeners' behaviors, and 3) conversational head generation to
integrate the talking/listening ability in one interlocutor. Along with the
datasets, we also propose corresponding baseline solutions to the three
aforementioned tasks. Experimental results show that our baseline method could
generate responsive and vivid agents that can collaborate with real person to
fulfil the whole conversation. Project page: https://vico.solutions/.