Hypergraph-based Multi-View Action Recognition using Event Cameras
Yue Gao Jiaxuan Lu Siqi Li Yipeng Li Shaoyi Du
Action recognition from video data forms a cornerstone with wide-ranging
applications. Single-view action recognition faces limitations due to its
reliance on a single viewpoint. In contrast, multi-view approaches capture
complementary information from various viewpoints for improved accuracy.
Recently, event cameras have emerged as innovative bio-inspired sensors,
leading to advancements in event-based action recognition. However, existing
works predominantly focus on single-view scenarios, leaving a gap in multi-view
event data exploitation, particularly in challenges like information deficit
and semantic misalignment. To bridge this gap, we introduce HyperMV, a
multi-view event-based action recognition framework. HyperMV converts discrete
event data into frame-like representations and extracts view-related features
using a shared convolutional network. By treating segments as vertices and
constructing hyperedges using rule-based and KNN-based strategies, a multi-view
hypergraph neural network that captures relationships across viewpoint and
temporal features is established. The vertex attention hypergraph propagation
is also introduced for enhanced feature fusion. To prompt research in this
area, we present the largest multi-view event-based action dataset
$\text{THU}^{\text{MV-EACT}}\text{-50}$, comprising 50 actions from 6
viewpoints, which surpasses existing datasets by over tenfold. Experimental
results show that HyperMV significantly outperforms baselines in both
cross-subject and cross-view scenarios, and also exceeds the state-of-the-arts
in frame-based multi-view action recognition.