Hierarchical Semantic Perceptual Listener Head Video Generation: A
  High-performance Pipeline
Zhigang Chang Weitai Hu Qing Yang Shibao Zheng
In dyadic speaker-listener interactions, the listener's head reactions along
with the speaker's head movements, constitute an important non-verbal semantic
expression together. The listener Head generation task aims to synthesize
responsive listener's head videos based on audios of the speaker and reference
images of the listener. Compared to the Talking-head generation, it is more
challenging to capture the correlation clues from the speaker's audio and
visual information. Following the ViCo baseline scheme, we propose a
high-performance solution by enhancing the hierarchical semantic extraction
capability of the audio encoder module and improving the decoder part, renderer
and post-processing modules. Our solution gets the first place on the official
leaderboard for the track of listening head generation. This paper is a
technical report of ViCo@2023 Conversational Head Generation Challenge in ACM
Multimedia 2023 conference.