Efficient Video Object Segmentation via Modulated Cross-Attention Memory
Abdelrahman Shaker Syed Talal Wasim Martin Danelljan Salman Khan Ming-Hsuan Yang Fahad Shahbaz Khan
Recently, transformer-based approaches have shown promising results for
semi-supervised video object segmentation. However, these approaches typically
struggle on long videos due to increased GPU memory demands, as they frequently
expand the memory bank every few frames. We propose a transformer-based
approach, named MAVOS, that introduces an optimized and dynamic long-term
modulated cross-attention (MCA) memory to model temporal smoothness without
requiring frequent memory expansion. The proposed MCA effectively encodes both
local and global features at various levels of granularity while efficiently
maintaining consistent speed regardless of the video length. Extensive
experiments on multiple benchmarks, LVOS, Long-Time Video, and DAVIS 2017,
demonstrate the effectiveness of our proposed contributions leading to
real-time inference and markedly reduced memory demands without any degradation
in segmentation accuracy on long videos. Compared to the best existing
transformer-based approach, our MAVOS increases the speed by 7.6x, while
significantly reducing the GPU memory by 87% with comparable segmentation
performance on short and long video datasets. Notably on the LVOS dataset, our
MAVOS achieves a J&F score of 63.3% while operating at 37 frames per second
(FPS) on a single V100 GPU. Our code and models will be publicly available at:
https://github.com/Amshaker/MAVOS.