Audio-Visual Glance Network for Efficient Video Recognition
Muhammad Adi Nugroho Sangmin Woo Sumin Lee Changick Kim
Deep learning has made significant strides in video understanding tasks, but
the computation required to classify lengthy and massive videos using
clip-level video classifiers remains impractical and prohibitively expensive.
To address this issue, we propose Audio-Visual Glance Network (AVGN), which
leverages the commonly available audio and visual modalities to efficiently
process the spatio-temporally important parts of a video. AVGN firstly divides
the video into snippets of image-audio clip pair and employs lightweight
unimodal encoders to extract global visual features and audio features. To
identify the important temporal segments, we use an Audio-Visual Temporal
Saliency Transformer (AV-TeST) that estimates the saliency scores of each
frame. To further increase efficiency in the spatial dimension, AVGN processes
only the important patches instead of the whole images. We use an
Audio-Enhanced Spatial Patch Attention (AESPA) module to produce a set of
enhanced coarse visual features, which are fed to a policy network that
produces the coordinates of the important patches. This approach enables us to
focus only on the most important spatio-temporally parts of the video, leading
to more efficient video recognition. Moreover, we incorporate various training
techniques and multi-modal feature fusion to enhance the robustness and
effectiveness of our AVGN. By combining these strategies, our AVGN sets new
state-of-the-art performance in multiple video recognition benchmarks while
achieving faster processing speed.