Language-Guided Diffusion Model for Visual Grounding
Sijia Chen Baochun Li
Visual grounding (VG) tasks involve explicit cross-modal alignment, as
semantically corresponding image regions are to be located for the language
phrases provided. Existing approaches complete such visual-text reasoning in a
single-step manner. Their performance causes high demands on large-scale
anchors and over-designed multi-modal fusion modules based on human priors,
leading to complicated frameworks that may be difficult to train and overfit to
specific scenarios. Even worse, such once-for-all reasoning mechanisms are
incapable of refining boxes continuously to enhance query-region matching. In
contrast, in this paper, we formulate an iterative reasoning process by
denoising diffusion modeling. Specifically, we propose a language-guided
diffusion framework for visual grounding, LG-DVG, which trains the model to
progressively reason queried object boxes by denoising a set of noisy boxes
with the language guide. To achieve this, LG-DVG gradually perturbs
query-aligned ground truth boxes to noisy ones and reverses this process step
by step, conditional on query semantics. Extensive experiments for our proposed
framework on five widely used datasets validate the superior performance of
solving visual grounding, a cross-modal alignment task, in a generative way.
The source codes are available at
\url{https://github.com/iQua/vgbase/tree/DiffusionVG}.