Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation
Yichen Yuan Yifan Wang Lijun Wang Xiaoqi Zhao Huchuan Lu Yu Wang Weibo Su Lei Zhang
Recent leading zero-shot video object segmentation (ZVOS) works devote to
integrating appearance and motion information by elaborately designing feature
fusion modules and identically applying them in multiple feature stages. Our
preliminary experiments show that with the strong long-range dependency
modeling capacity of Transformer, simply concatenating the two modality
features and feeding them to vanilla Transformers for feature fusion can
distinctly benefit the performance but at a cost of heavy computation. Through
further empirical analysis, we find that attention dependencies learned in
Transformer in different stages exhibit completely different properties: global
query-independent dependency in the low-level stages and semantic-specific
dependency in the high-level stages. Motivated by the observations, we propose
two Transformer variants: i) Context-Sharing Transformer (CST) that learns the
global-shared contextual information within image frames with a lightweight
computation. ii) Semantic Gathering-Scattering Transformer (SGST) that models
the semantic correlation separately for the foreground and background and
reduces the computation cost with a soft token merging mechanism. We apply CST
and SGST for low-level and high-level feature fusions, respectively,
formulating a level-isomerous Transformer framework for ZVOS task. Compared
with the baseline that uses vanilla Transformers for multi-stage fusion, ours
significantly increase the speed by 13 times and achieves new state-of-the-art
ZVOS performance. Code is available at https://github.com/DLUT-yyc/Isomer.