Spatio-temporal Prompting Network for Robust Video Feature Extraction
Guanxiong Sun Chi Wang Zhaoyu Zhang Jiankang Deng Stefanos Zafeiriou Yang Hua
Frame quality deterioration is one of the main challenges in the field of
video understanding. To compensate for the information loss caused by
deteriorated frames, recent approaches exploit transformer-based integration
modules to obtain spatio-temporal information. However, these integration
modules are heavy and complex. Furthermore, each integration module is
specifically tailored for its target task, making it difficult to generalise to
multiple tasks. In this paper, we present a neat and unified framework, called
Spatio-Temporal Prompting Network (STPN). It can efficiently extract robust and
accurate video features by dynamically adjusting the input features in the
backbone network. Specifically, STPN predicts several video prompts containing
spatio-temporal information of neighbour frames. Then, these video prompts are
prepended to the patch embeddings of the current frame as the updated input for
video feature extraction. Moreover, STPN is easy to generalise to various video
tasks because it does not contain task-specific modules. Without bells and
whistles, STPN achieves state-of-the-art performance on three widely-used
datasets for different video understanding tasks, i.e., ImageNetVID for video
object detection, YouTubeVIS for video instance segmentation, and GOT-10k for
visual object tracking. Code is available at
https://github.com/guanxiongsun/vfe.pytorch.