MARIS: Referring Image Segmentation via Mutual-Aware Attention Features
Mengxi Zhang Yiming Liu Xiangjun Yin Huanjing Yue Jingyu Yang
Referring image segmentation (RIS) aims to segment a particular region based
on a language expression prompt. Existing methods incorporate linguistic
features into visual features and obtain multi-modal features for mask
decoding. However, these methods may segment the visually salient entity
instead of the correct referring region, as the multi-modal features are
dominated by the abundant visual context. In this paper, we propose MARIS, a
referring image segmentation method that leverages the Segment Anything Model
(SAM) and introduces a mutual-aware attention mechanism to enhance the
cross-modal fusion via two parallel branches. Specifically, our mutual-aware
attention mechanism consists of Vision-Guided Attention and Language-Guided
Attention, which bidirectionally model the relationship between visual and
linguistic features. Correspondingly, we design a Mask Decoder to enable
explicit linguistic guidance for more consistent segmentation with the language
expression. To this end, a multi-modal query token is proposed to integrate
linguistic information and interact with visual information simultaneously.
Extensive experiments on three benchmark datasets show that our method
outperforms the state-of-the-art RIS methods. Our code will be publicly
available.