A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human
  Interaction Recognition
Ruoqi Yin Jianqin Yin
Human Interaction Recognition is the process of identifying interactive
actions between multiple participants in a specific situation. The aim is to
recognise the action interactions between multiple entities and their meaning.
Many single Convolutional Neural Network has issues, such as the inability to
capture global instance interaction features or difficulty in training, leading
to ambiguity in action semantics. In addition, the computational complexity of
the Transformer cannot be ignored, and its ability to capture local information
and motion features in the image is poor. In this work, we propose a Two-stream
Hybrid CNN-Transformer Network (THCT-Net), which exploits the local specificity
of CNN and models global dependencies through the Transformer. CNN and
Transformer simultaneously model the entity, time and space relationships
between interactive entities respectively. Specifically, Transformer-based
stream integrates 3D convolutions with multi-head self-attention to learn
inter-token correlations; We propose a new multi-branch CNN framework for
CNN-based streams that automatically learns joint spatio-temporal features from
skeleton sequences. The convolutional layer independently learns the local
features of each joint neighborhood and aggregates the features of all joints.
And the raw skeleton coordinates as well as their temporal difference are
integrated with a dual-branch paradigm to fuse the motion features of the
skeleton. Besides, a residual structure is added to speed up training
convergence. Finally, the recognition results of the two branches are fused
using parallel splicing. Experimental results on diverse and challenging
datasets, demonstrate that the proposed method can better comprehend and infer
the meaning and context of various actions, outperforming state-of-the-art
methods.