Using an LLM to Turn Sign Spottings into Spoken Language Sentences
Ozge Mercanoglu Sincan Necati Cihan Camgoz Richard Bowden
Sign Language Translation (SLT) is a challenging task that aims to generate
spoken language sentences from sign language videos. In this paper, we
introduce a hybrid SLT approach, Spotter+GPT, that utilizes a sign spotter and
a pretrained large language model to improve SLT performance. Our method builds
upon the strengths of both components. The videos are first processed by the
spotter, which is trained on a linguistic sign language dataset, to identify
individual signs. These spotted signs are then passed to the powerful language
model, which transforms them into coherent and contextually appropriate spoken
language sentences.