Temporally-Adaptive Models for Efficient Video Understanding
Ziyuan Huang Shiwei Zhang Liang Pan Zhiwu Qing Yingya Zhang Ziwei Liu Marcelo H. Ang Jr
Spatial convolutions are extensively used in numerous deep video models. It
fundamentally assumes spatio-temporal invariance, i.e., using shared weights
for every location in different frames. This work presents Temporally-Adaptive
Convolutions (TAdaConv) for video understanding, which shows that adaptive
weight calibration along the temporal dimension is an efficient way to
facilitate modeling complex temporal dynamics in videos. Specifically, TAdaConv
empowers spatial convolutions with temporal modeling abilities by calibrating
the convolution weights for each frame according to its local and global
temporal context. Compared to existing operations for temporal modeling,
TAdaConv is more efficient as it operates over the convolution kernels instead
of the features, whose dimension is an order of magnitude smaller than the
spatial resolutions. Further, kernel calibration brings an increased model
capacity. Based on this readily plug-in operation TAdaConv as well as its
extension, i.e., TAdaConvV2, we construct TAdaBlocks to empower ConvNeXt and
Vision Transformer to have strong temporal modeling capabilities. Empirical
results show TAdaConvNeXtV2 and TAdaFormer perform competitively against
state-of-the-art convolutional and Transformer-based models in various video
understanding benchmarks. Our codes and models are released at:
https://github.com/alibaba-mmai-research/TAdaConv.