Rethinking Referring Object Removal
Xiangtian Xue Jiasong Wu Youyong Kong Lotfi Senhadji Huazhong Shu
Referring object removal refers to removing the specific object in an image
referred by natural language expressions and filling the missing region with
reasonable semantics. To address this task, we construct the ComCOCO, a
synthetic dataset consisting of 136,495 referring expressions for 34,615
objects in 23,951 image pairs. Each pair contains an image with referring
expressions and the ground truth after elimination. We further propose an
end-to-end syntax-aware hybrid mapping network with an encoding-decoding
structure. Linguistic features are hierarchically extracted at the syntactic
level and fused in the downsampling process of visual features with multi-head
attention. The feature-aligned pyramid network is leveraged to generate
segmentation masks and replace internal pixels with region affinity learned
from external semantics in high-level feature maps. Extensive experiments
demonstrate that our model outperforms diffusion models and two-stage methods
which process the segmentation and inpainting task separately by a significant
margin.