BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed
  Dual-Branch Diffusion
Xuan Ju Xian Liu Xintao Wang Yuxuan Bian Ying Shan Qiang Xu
Image inpainting, the process of restoring corrupted images, has seen
significant advancements with the advent of diffusion models (DMs). Despite
these advancements, current DM adaptations for inpainting, which involve
modifications to the sampling strategy or the development of
inpainting-specific DMs, frequently suffer from semantic inconsistencies and
reduced image quality. Addressing these challenges, our work introduces a novel
paradigm: the division of masked image features and noisy latent into separate
branches. This division dramatically diminishes the model's learning load,
facilitating a nuanced incorporation of essential masked image information in a
hierarchical fashion. Herein, we present BrushNet, a novel plug-and-play
dual-branch model engineered to embed pixel-level masked image features into
any pre-trained DM, guaranteeing coherent and enhanced image inpainting
outcomes. Additionally, we introduce BrushData and BrushBench to facilitate
segmentation-based inpainting training and performance assessment. Our
extensive experimental analysis demonstrates BrushNet's superior performance
over existing models across seven key metrics, including image quality, mask
region preservation, and textual coherence.