Diffusion Models are Geometry Critics: Single Image 3D Editing Using
  Pre-Trained Diffusion Priors
Ruicheng Wang Jianfeng Xiang Jiaolong Yang Xin Tong
We propose a novel image editing technique that enables 3D manipulations on
single images, such as object rotation and translation. Existing 3D-aware image
editing approaches typically rely on synthetic multi-view datasets for training
specialized models, thus constraining their effectiveness on open-domain images
featuring significantly more varied layouts and styles. In contrast, our method
directly leverages powerful image diffusion models trained on a broad spectrum
of text-image pairs and thus retain their exceptional generalization abilities.
This objective is realized through the development of an iterative novel view
synthesis and geometry alignment algorithm. The algorithm harnesses diffusion
models for dual purposes: they provide appearance prior by predicting novel
views of the selected object using estimated depth maps, and they act as a
geometry critic by correcting misalignments in 3D shapes across the sampled
views. Our method can generate high-quality 3D-aware image edits with large
viewpoint transformations and high appearance and shape consistency with the
input image, pushing the boundaries of what is possible with single-image
3D-aware editing.