Probing Language Models' Gesture Understanding for Enhanced Human-AI
  Interaction
Philipp Wicke
The rise of Large Language Models (LLMs) has affected various disciplines
that got beyond mere text generation. Going beyond their textual nature, this
project proposal aims to investigate the interaction between LLMs and
non-verbal communication, specifically focusing on gestures. The proposal sets
out a plan to examine the proficiency of LLMs in deciphering both explicit and
implicit non-verbal cues within textual prompts and their ability to associate
these gestures with various contextual factors. The research proposes to test
established psycholinguistic study designs to construct a comprehensive dataset
that pairs textual prompts with detailed gesture descriptions, encompassing
diverse regional variations, and semantic labels. To assess LLMs' comprehension
of gestures, experiments are planned, evaluating their ability to simulate
human behaviour in order to replicate psycholinguistic experiments. These
experiments consider cultural dimensions and measure the agreement between
LLM-identified gestures and the dataset, shedding light on the models'
contextual interpretation of non-verbal cues (e.g. gestures).