Multi-View Unsupervised Image Generation with Cross Attention Guidance
Llukman Cerkezi Aram Davtyan Sepehr Sameni Paolo Favaro
The growing interest in novel view synthesis, driven by Neural Radiance Field
(NeRF) models, is hindered by scalability issues due to their reliance on
precisely annotated multi-view images. Recent models address this by
fine-tuning large text2image diffusion models on synthetic multi-view data.
Despite robust zero-shot generalization, they may need post-processing and can
face quality issues due to the synthetic-real domain gap. This paper introduces
a novel pipeline for unsupervised training of a pose-conditioned diffusion
model on single-category datasets. With the help of pretrained self-supervised
Vision Transformers (DINOv2), we identify object poses by clustering the
dataset through comparing visibility and locations of specific object parts.
The pose-conditioned diffusion model, trained on pose labels, and equipped with
cross-frame attention at inference time ensures cross-view consistency, that is
further aided by our novel hard-attention guidance. Our model, MIRAGE,
surpasses prior work in novel view synthesis on real images. Furthermore,
MIRAGE is robust to diverse textures and geometries, as demonstrated with our
experiments on synthetic images generated with pretrained Stable Diffusion.