Dynamic Spatial-Temporal Aggregation for Skeleton-Aware Sign Language
  Recognition
Lianyu Hu Liqing Gao Zekang Liu Wei Feng
Skeleton-aware sign language recognition (SLR) has gained popularity due to
its ability to remain unaffected by background information and its lower
computational requirements. Current methods utilize spatial graph modules and
temporal modules to capture spatial and temporal features, respectively.
However, their spatial graph modules are typically built on fixed graph
structures such as graph convolutional networks or a single learnable graph,
which only partially explore joint relationships. Additionally, a simple
temporal convolution kernel is used to capture temporal information, which may
not fully capture the complex movement patterns of different signers. To
overcome these limitations, we propose a new spatial architecture consisting of
two concurrent branches, which build input-sensitive joint relationships and
incorporates specific domain knowledge for recognition, respectively. These two
branches are followed by an aggregation process to distinguishe important joint
connections. We then propose a new temporal module to model multi-scale
temporal information to capture complex human dynamics. Our method achieves
state-of-the-art accuracy compared to previous skeleton-aware methods on four
large-scale SLR benchmarks. Moreover, our method demonstrates superior accuracy
compared to RGB-based methods in most cases while requiring much fewer
computational resources, bringing better accuracy-computation trade-off. Code
is available at https://github.com/hulianyuyy/DSTA-SLR.