GOEnFusion: Gradient Origin Encodings for 3D Forward Diffusion Models
Animesh Karnewar Andrea Vedaldi Niloy J. Mitra David Novotny
The recently introduced Forward-Diffusion method allows to train a 3D
diffusion model using only 2D images for supervision. However, it does not
easily generalise to different 3D representations and requires a
computationally expensive auto-regressive sampling process to generate the
underlying 3D scenes. In this paper, we propose GOEn: Gradient Origin Encoding
(pronounced "gone"). GOEn can encode input images into any type of 3D
representation without the need to use a pre-trained image feature extractor.
It can also handle single, multiple or no source view(s) alike, by design, and
tries to maximise the information transfer from the views to the encodings. Our
proposed GOEnFusion model pairs GOEn encodings with a realisation of the
Forward-Diffusion model which addresses the limitations of the vanilla
Forward-Diffusion realisation. We evaluate how much information the GOEn
mechanism transfers to the encoded representations, and how well it captures
the prior distribution over the underlying 3D scenes, through the lens of a
partial AutoEncoder. Lastly, the efficacy of the GOEnFusion model is evaluated
on the recently proposed OmniObject3D dataset while comparing to the
state-of-the-art Forward and non-Forward-Diffusion models and other 3D
generative models.