SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by
  Visual-Textual Contrastive Learning
Hao Chen Jiaze Wang Ziyu Guo Jinpeng Li Donghao Zhou Bian Wu Chenyong Guan Guangyong Chen Pheng-Ann Heng
Sign language recognition (SLR) plays a vital role in facilitating
communication for the hearing-impaired community. SLR is a weakly supervised
task where entire videos are annotated with glosses, making it challenging to
identify the corresponding gloss within a video segment. Recent studies
indicate that the main bottleneck in SLR is the insufficient training caused by
the limited availability of large-scale datasets. To address this challenge, we
present SignVTCL, a multi-modal continuous sign language recognition framework
enhanced by visual-textual contrastive learning, which leverages the full
potential of multi-modal data and the generalization ability of language model.
SignVTCL integrates multi-modal data (video, keypoints, and optical flow)
simultaneously to train a unified visual backbone, thereby yielding more robust
visual representations. Furthermore, SignVTCL contains a visual-textual
alignment approach incorporating gloss-level and sentence-level alignment to
ensure precise correspondence between visual features and glosses at the level
of individual glosses and sentence. Experimental results conducted on three
datasets, Phoenix-2014, Phoenix-2014T, and CSL-Daily, demonstrate that SignVTCL
achieves state-of-the-art results compared with previous methods.