Differentiable Resolution Compression and Alignment for Efficient Video
  Classification and Retrieval
Rui Deng Qian Wu Yuke Li Haoran Fu
Optimizing video inference efficiency has become increasingly important with
the growing demand for video analysis in various fields. Some existing methods
achieve high efficiency by explicit discard of spatial or temporal information,
which poses challenges in fast-changing and fine-grained scenarios. To address
these issues, we propose an efficient video representation network with
Differentiable Resolution Compression and Alignment mechanism, which compresses
non-essential information in the early stage of the network to reduce
computational costs while maintaining consistent temporal correlations.
Specifically, we leverage a Differentiable Context-aware Compression Module to
encode the saliency and non-saliency frame features, refining and updating the
features into a high-low resolution video sequence. To process the new
sequence, we introduce a new Resolution-Align Transformer Layer to capture
global temporal correlations among frame features with different resolutions,
while reducing spatial computation costs quadratically by utilizing fewer
spatial tokens in low-resolution non-saliency frames. The entire network can be
end-to-end optimized via the integration of the differentiable compression
module. Experimental results show that our method achieves the best trade-off
between efficiency and performance on near-duplicate video retrieval and
competitive results on dynamic video classification compared to
state-of-the-art methods. Code:https://github.com/dun-research/DRCA