SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained
  Object Insertion and Layout Control
Jaskirat Singh Jianming Zhang Qing Liu Cameron Smith Zhe Lin Liang Zheng
The field of generative image inpainting and object insertion has made
significant progress with the recent advent of latent diffusion models.
Utilizing a precise object mask can greatly enhance these applications.
However, due to the challenges users encounter in creating high-fidelity masks,
there is a tendency for these methods to rely on more coarse masks (e.g.,
bounding box) for these applications. This results in limited control and
compromised background content preservation. To overcome these limitations, we
introduce SmartMask, which allows any novice user to create detailed masks for
precise object insertion. Combined with a ControlNet-Inpaint model, our
experiments demonstrate that SmartMask achieves superior object insertion
quality, preserving the background content more effectively than previous
methods. Notably, unlike prior works the proposed approach can also be used
even without user-mask guidance, which allows it to perform mask-free object
insertion at diverse positions and scales. Furthermore, we find that when used
iteratively with a novel instruction-tuning based planning model, SmartMask can
be used to design detailed layouts from scratch. As compared with user-scribble
based layout design, we observe that SmartMask allows for better quality
outputs with layout-to-image generation methods. Project page is available at
https://smartmask-gen.github.io