MK-SGN: A Spiking Graph Convolutional Network with Multimodal Fusion and
  Knowledge Distillation for Skeleton-based Action Recognition
Naichuan Zheng Hailun Xia Zeyu Liang
In recent years, skeleton-based action recognition, leveraging multimodal
Graph Convolutional Networks (GCN), has achieved remarkable results. However,
due to their deep structure and reliance on continuous floating-point
operations, GCN-based methods are energy-intensive. To address this issue, we
propose an innovative Spiking Graph Convolutional Network with Multimodal
Fusion and Knowledge Distillation (MK-SGN). By merging the energy efficiency of
Spiking Neural Network (SNN) with the graph representation capability of GCN,
the proposed MK-SGN reduces energy consumption while maintaining recognition
accuracy. Firstly, we convert GCN into Spiking Graph Convolutional Network
(SGN) and construct a foundational Base-SGN for skeleton-based action
recognition, establishing a new benchmark and paving the way for future
research exploration. Secondly, we further propose a Spiking Multimodal Fusion
module (SMF), leveraging mutual information to process multimodal data more
efficiently. Additionally, we introduce a spiking attention mechanism and
design a Spatio Graph Convolution module with a Spatial Global Spiking
Attention mechanism (SA-SGC), enhancing feature learning capability.
Furthermore, we delve into knowledge distillation methods from multimodal GCN
to SGN and propose a novel, integrated method that simultaneously focuses on
both intermediate layer distillation and soft label distillation to improve the
performance of SGN. On two challenging datasets for skeleton-based action
recognition, MK-SGN outperforms the state-of-the-art GCN-like frameworks in
reducing computational load and energy consumption. In contrast, typical GCN
methods typically consume more than 35mJ per action sample, while MK-SGN
reduces energy consumption by more than 98%.