PRISM: Progressive Restoration for Scene Graph-based Image Manipulation
Pavel Jahoda Azade Farshad Yousef Yeganeh Ehsan Adeli Nassir Navab
Scene graphs have emerged as accurate descriptive priors for image generation
and manipulation tasks, however, their complexity and diversity of the shapes
and relations of objects in data make it challenging to incorporate them into
the models and generate high-quality results. To address these challenges, we
propose PRISM, a novel progressive multi-head image manipulation approach to
improve the accuracy and quality of the manipulated regions in the scene. Our
image manipulation framework is trained using an end-to-end denoising masked
reconstruction proxy task, where the masked regions are progressively unmasked
from the outer regions to the inner part. We take advantage of the outer part
of the masked area as they have a direct correlation with the context of the
scene. Moreover, our multi-head architecture simultaneously generates detailed
object-specific regions in addition to the entire image to produce
higher-quality images. Our model outperforms the state-of-the-art methods in
the semantic image manipulation task on the CLEVR and Visual Genome datasets.
Our results demonstrate the potential of our approach for enhancing the quality
and precision of scene graph-based image manipulation.