InstructGIE: Towards Generalizable Image Editing
Zichong Meng Changdi Yang Jun Liu Hao Tang Pu Zhao Yanzhi Wang
Recent advances in image editing have been driven by the development of
denoising diffusion models, marking a significant leap forward in this field.
Despite these advances, the generalization capabilities of recent image editing
approaches remain constrained. In response to this challenge, our study
introduces a novel image editing framework with enhanced generalization
robustness by boosting in-context learning capability and unifying language
instruction. This framework incorporates a module specifically optimized for
image editing tasks, leveraging the VMamba Block and an editing-shift matching
strategy to augment in-context learning. Furthermore, we unveil a selective
area-matching technique specifically engineered to address and rectify
corrupted details in generated images, such as human facial features, to
further improve the quality. Another key innovation of our approach is the
integration of a language unification technique, which aligns language
embeddings with editing semantics to elevate the quality of image editing.
Moreover, we compile the first dataset for image editing with visual prompts
and editing instructions that could be used to enhance in-context capability.
Trained on this dataset, our methodology not only achieves superior synthesis
quality for trained tasks, but also demonstrates robust generalization
capability across unseen vision tasks through tailored prompts.