Improving Continuous Sign Language Recognition with Adapted Image Models
Lianyu Hu Tongkai Shi Liqing Gao Zekang Liu Wei Feng
The increase of web-scale weakly labelled image-text pairs have greatly
facilitated the development of large-scale vision-language models (e.g., CLIP),
which have shown impressive generalization performance over a series of
downstream tasks. However, the massive model size and scarcity of available
data limit their applications to fine-tune the whole model in downstream tasks.
Besides, fully fine-tuning the model easily forgets the generic essential
knowledge acquired in the pretraining stage and overfits the downstream data.
To enable high efficiency when adapting these large vision-language models
(e.g., CLIP) to performing continuous sign language recognition (CSLR) while
preserving their generalizability, we propose a novel strategy (AdaptSign).
Especially, CLIP is adopted as the visual backbone to extract frame-wise
features whose parameters are fixed, and a set of learnable modules are
introduced to model spatial sign variations or capture temporal sign movements.
The introduced additional modules are quite lightweight, only owning 3.2% extra
computations with high efficiency. The generic knowledge acquired in the
pretraining stage is well-preserved in the frozen CLIP backbone in this
process. Extensive experiments show that despite being efficient, AdaptSign is
able to demonstrate superior performance across a series of CSLR benchmarks
including PHOENIX14, PHOENIX14-T, CSL-Daily and CSL compared to existing
methods. Visualizations show that AdaptSign could learn to dynamically pay
major attention to the informative spatial regions and cross-frame trajectories
in sign videos.