Can Language Models Learn to Listen?
Evonne Ng Sanjay Subramanian Dan Klein Angjoo Kanazawa Trevor Darrell Shiry Ginosar
We present a framework for generating appropriate facial responses from a
listener in dyadic social interactions based on the speaker's words. Given an
input transcription of the speaker's words with their timestamps, our approach
autoregressively predicts a response of a listener: a sequence of listener
facial gestures, quantized using a VQ-VAE. Since gesture is a language
component, we propose treating the quantized atomic motion elements as
additional language token inputs to a transformer-based large language model.
Initializing our transformer with the weights of a language model pre-trained
only on text results in significantly higher quality listener responses than
training a transformer from scratch. We show that our generated listener motion
is fluent and reflective of language semantics through quantitative metrics and
a qualitative user study. In our evaluation, we analyze the model's ability to
utilize temporal and semantic aspects of spoken text. Project page:
https://people.eecs.berkeley.edu/~evonne_ng/projects/text2listen/