VK-G2T: Vision and Context Knowledge enhanced Gloss2Text
Liqiang Jing Xuemeng Song Xinxing Zu Na Zheng Zhongzhou Zhao Liqiang Nie
Existing sign language translation methods follow a two-stage pipeline: first
converting the sign language video to a gloss sequence (i.e. Sign2Gloss) and
then translating the generated gloss sequence into a spoken language sentence
(i.e. Gloss2Text). While previous studies have focused on boosting the
performance of the Sign2Gloss stage, we emphasize the optimization of the
Gloss2Text stage. However, this task is non-trivial due to two distinct
features of Gloss2Text: (1) isolated gloss input and (2) low-capacity gloss
vocabulary. To address these issues, we propose a vision and context knowledge
enhanced Gloss2Text model, named VK-G2T, which leverages the visual content of
the sign language video to learn the properties of the target sentence and
exploit the context knowledge to facilitate the adaptive translation of gloss
words. Extensive experiments conducted on a Chinese benchmark validate the
superiority of our model.