Joint Modeling of Feature, Correspondence, and a Compressed Memory for
  Video Object Segmentation
Jiaming Zhang Yutao Cui Gangshan Wu Limin Wang
Current prevailing Video Object Segmentation (VOS) methods usually perform
dense matching between the current and reference frames after extracting their
features. One on hand, the decoupled modeling restricts the targets information
propagation only at high-level feature space. On the other hand, the pixel-wise
matching leads to a lack of holistic understanding of the targets. To overcome
these issues, we propose a unified VOS framework, coined as JointFormer, for
joint modeling the three elements of feature, correspondence, and a compressed
memory. The core design is the Joint Block, utilizing the flexibility of
attention to simultaneously extract feature and propagate the targets
information to the current tokens and the compressed memory token. This scheme
allows to perform extensive information propagation and discriminative feature
learning. To incorporate the long-term temporal targets information, we also
devise a customized online updating mechanism for the compressed memory token,
which can prompt the information flow along the temporal dimension and thus
improve the global modeling capability. Under the design, our method achieves a
new state-of-art performance on DAVIS 2017 val/test-dev (89.7% and 87.6%) and
YouTube-VOS 2018/2019 val (87.0% and 87.0%) benchmarks, outperforming existing
works by a large margin.