Text Augmented Spatial-aware Zero-shot Referring Image Segmentation
Yucheng Suo Linchao Zhu Yi Yang
In this paper, we study a challenging task of zero-shot referring image
segmentation. This task aims to identify the instance mask that is most related
to a referring expression without training on pixel-level annotations. Previous
research takes advantage of pre-trained cross-modal models, e.g., CLIP, to
align instance-level masks with referring expressions. %Yet, CLIP only
considers image-text pair level alignment, which neglects fine-grained image
region and complex sentence matching. Yet, CLIP only considers the global-level
alignment of image-text pairs, neglecting fine-grained matching between the
referring sentence and local image regions. To address this challenge, we
introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image
segmentation framework that is training-free and robust to various visual
encoders. TAS incorporates a mask proposal network for instance-level mask
extraction, a text-augmented visual-text matching score for mining the
image-text correlation, and a spatial rectifier for mask post-processing.
Notably, the text-augmented visual-text matching score leverages a $P$ score
and an $N$-score in addition to the typical visual-text matching score. The
$P$-score is utilized to close the visual-text domain gap through a surrogate
captioning model, where the score is computed between the surrogate
model-generated texts and the referring expression. The $N$-score considers the
fine-grained alignment of region-text pairs via negative phrase mining,
encouraging the masked image to be repelled from the mined distracting phrases.
Extensive experiments are conducted on various datasets, including RefCOCO,
RefCOCO+, and RefCOCOg. The proposed method clearly outperforms
state-of-the-art zero-shot referring image segmentation methods.