VPUFormer: Visual Prompt Unified Transformer for Interactive Image
  Segmentation
Xu Zhang Kailun Yang Jiacheng Lin Jin Yuan Zhiyong Li Shutao Li
The integration of diverse visual prompts like clicks, scribbles, and boxes
in interactive image segmentation could significantly facilitate user
interaction as well as improve interaction efficiency. Most existing studies
focus on a single type of visual prompt by simply concatenating prompts and
images as input for segmentation prediction, which suffers from low-efficiency
prompt representation and weak interaction issues. This paper proposes a simple
yet effective Visual Prompt Unified Transformer (VPUFormer), which introduces a
concise unified prompt representation with deeper interaction to boost the
segmentation performance. Specifically, we design a Prompt-unified Encoder
(PuE) by using Gaussian mapping to generate a unified one-dimensional vector
for click, box, and scribble prompts, which well captures users' intentions as
well as provides a denser representation of user prompts. In addition, we
present a Prompt-to-Pixel Contrastive Loss (P2CL) that leverages user feedback
to gradually refine candidate semantic features, aiming to bring image semantic
features closer to the features that are similar to the user prompt, while
pushing away those image semantic features that are dissimilar to the user
prompt, thereby correcting results that deviate from expectations. On this
basis, our approach injects prompt representations as queries into Dual-cross
Merging Attention (DMA) blocks to perform a deeper interaction between image
and query inputs. A comprehensive variety of experiments on seven challenging
datasets demonstrates that the proposed VPUFormer with PuE, DMA, and P2CL
achieves consistent improvements, yielding state-of-the-art segmentation
performance. Our code will be made publicly available at
https://github.com/XuZhang1211/VPUFormer.