Fully Transformer-Equipped Architecture for End-to-End Referring Video
  Object Segmentation
Ping Li Yu Zhang Li Yuan Xianghua Xu
Referring Video Object Segmentation (RVOS) requires segmenting the object in
video referred by a natural language query. Existing methods mainly rely on
sophisticated pipelines to tackle such cross-modal task, and do not explicitly
model the object-level spatial context which plays an important role in
locating the referred object. Therefore, we propose an end-to-end RVOS
framework completely built upon transformers, termed \textit{Fully
Transformer-Equipped Architecture} (FTEA), which treats the RVOS task as a mask
sequence learning problem and regards all the objects in video as candidate
objects. Given a video clip with a text query, the visual-textual features are
yielded by encoder, while the corresponding pixel-level and word-level features
are aligned in terms of semantic similarity. To capture the object-level
spatial context, we have developed the Stacked Transformer, which individually
characterizes the visual appearance of each candidate object, whose feature map
is decoded to the binary mask sequence in order directly. Finally, the model
finds the best matching between mask sequence and text query. In addition, to
diversify the generated masks for candidate objects, we impose a diversity loss
on the model for capturing more accurate mask of the referred object. Empirical
studies have shown the superiority of the proposed method on three benchmarks,
e.g., FETA achieves 45.1% and 38.7% in terms of mAP on A2D Sentences (3782
videos) and J-HMDB Sentences (928 videos), respectively; it achieves 56.6% in
terms of $\mathcal{J\&F}$ on Ref-YouTube-VOS (3975 videos and 7451 objects).
Particularly, compared to the best candidate method, it has a gain of 2.1% and
3.2% in terms of P$@$0.5 on the former two, respectively, while it has a gain
of 2.9% in terms of $\mathcal{J}$ on the latter one.