OneVOS: Unifying Video Object Segmentation with All-in-One Transformer
  Framework
Wanyun Li Pinxue Guo Xinyu Zhou Lingyi Hong Yangji He Xiangyu Zheng Wei Zhang Wenqiang Zhang
Contemporary Video Object Segmentation (VOS) approaches typically consist
stages of feature extraction, matching, memory management, and multiple objects
aggregation. Recent advanced models either employ a discrete modeling for these
components in a sequential manner, or optimize a combined pipeline through
substructure aggregation. However, these existing explicit staged approaches
prevent the VOS framework from being optimized as a unified whole, leading to
the limited capacity and suboptimal performance in tackling complex videos. In
this paper, we propose OneVOS, a novel framework that unifies the core
components of VOS with All-in-One Transformer. Specifically, to unify all
aforementioned modules into a vision transformer, we model all the features of
frames, masks and memory for multiple objects as transformer tokens, and
integrally accomplish feature extraction, matching and memory management of
multiple objects through the flexible attention mechanism. Furthermore, a
Unidirectional Hybrid Attention is proposed through a double decoupling of the
original attention operation, to rectify semantic errors and ambiguities of
stored tokens in OneVOS framework. Finally, to alleviate the storage burden and
expedite inference, we propose the Dynamic Token Selector, which unveils the
working mechanism of OneVOS and naturally leads to a more efficient version of
OneVOS. Extensive experiments demonstrate the superiority of OneVOS, achieving
state-of-the-art performance across 7 datasets, particularly excelling in
complex LVOS and MOSE datasets with 70.1% and 66.4% $J \& F$ scores, surpassing
previous state-of-the-art methods by 4.2% and 7.0%, respectively. And our code
will be available for reproducibility and further research.