SkateFormer: Skeletal-Temporal Transformer for Human Action Recognition
Jeonghyeok Do Munchurl Kim
Skeleton-based action recognition, which classifies human actions based on
the coordinates of joints and their connectivity within skeleton data, is
widely utilized in various scenarios. While Graph Convolutional Networks (GCNs)
have been proposed for skeleton data represented as graphs, they suffer from
limited receptive fields constrained by joint connectivity. To address this
limitation, recent advancements have introduced transformer-based methods.
However, capturing correlations between all joints in all frames requires
substantial memory resources. To alleviate this, we propose a novel approach
called Skeletal-Temporal Transformer (SkateFormer) that partitions joints and
frames based on different types of skeletal-temporal relation (Skate-Type) and
performs skeletal-temporal self-attention (Skate-MSA) within each partition. We
categorize the key skeletal-temporal relations for action recognition into a
total of four distinct types. These types combine (i) two skeletal relation
types based on physically neighboring and distant joints, and (ii) two temporal
relation types based on neighboring and distant frames. Through this
partition-specific attention strategy, our SkateFormer can selectively focus on
key joints and frames crucial for action recognition in an action-adaptive
manner with efficient computation. Extensive experiments on various benchmark
datasets validate that our SkateFormer outperforms recent state-of-the-art
methods.