Patch Spatio-Temporal Relation Prediction for Video Anomaly Detection
Hao Shen Lu Shi Wanru Xu Yigang Cen Linna Zhang Gaoyun An
Video Anomaly Detection (VAD), aiming to identify abnormalities within a
specific context and timeframe, is crucial for intelligent Video Surveillance
Systems. While recent deep learning-based VAD models have shown promising
results by generating high-resolution frames, they often lack competence in
preserving detailed spatial and temporal coherence in video frames. To tackle
this issue, we propose a self-supervised learning approach for VAD through an
inter-patch relationship prediction task. Specifically, we introduce a
two-branch vision transformer network designed to capture deep visual features
of video frames, addressing spatial and temporal dimensions responsible for
modeling appearance and motion patterns, respectively. The inter-patch
relationship in each dimension is decoupled into inter-patch similarity and the
order information of each patch. To mitigate memory consumption, we convert the
order information prediction task into a multi-label learning problem, and the
inter-patch similarity prediction task into a distance matrix regression
problem. Comprehensive experiments demonstrate the effectiveness of our method,
surpassing pixel-generation-based methods by a significant margin across three
public benchmarks. Additionally, our approach outperforms other self-supervised
learning-based methods.