A Transformer Model for Boundary Detection in Continuous Sign Language
Razieh Rastgoo Kourosh Kiani Sergio Escalera
Sign Language Recognition (SLR) has garnered significant attention from
researchers in recent years, particularly the intricate domain of Continuous
Sign Language Recognition (CSLR), which presents heightened complexity compared
to Isolated Sign Language Recognition (ISLR). One of the prominent challenges
in CSLR pertains to accurately detecting the boundaries of isolated signs
within a continuous video stream. Additionally, the reliance on handcrafted
features in existing models poses a challenge to achieving optimal accuracy. To
surmount these challenges, we propose a novel approach utilizing a
Transformer-based model. Unlike traditional models, our approach focuses on
enhancing accuracy while eliminating the need for handcrafted features. The
Transformer model is employed for both ISLR and CSLR. The training process
involves using isolated sign videos, where hand keypoint features extracted
from the input video are enriched using the Transformer model. Subsequently,
these enriched features are forwarded to the final classification layer. The
trained model, coupled with a post-processing method, is then applied to detect
isolated sign boundaries within continuous sign videos. The evaluation of our
model is conducted on two distinct datasets, including both continuous signs
and their corresponding isolated signs, demonstrates promising results.