GenN2N: Generative NeRF2NeRF Translation
Xiangyue Liu Han Xue Kunming Luo Ping Tan Li Yi
We present GenN2N, a unified NeRF-to-NeRF translation framework for various
NeRF translation tasks such as text-driven NeRF editing, colorization,
super-resolution, inpainting, etc. Unlike previous methods designed for
individual translation tasks with task-specific schemes, GenN2N achieves all
these NeRF editing tasks by employing a plug-and-play image-to-image translator
to perform editing in the 2D domain and lifting 2D edits into the 3D NeRF
space. Since the 3D consistency of 2D edits may not be assured, we propose to
model the distribution of the underlying 3D edits through a generative model
that can cover all possible edited NeRFs. To model the distribution of 3D
edited NeRFs from 2D edited images, we carefully design a VAE-GAN that encodes
images while decoding NeRFs. The latent space is trained to align with a
Gaussian distribution and the NeRFs are supervised through an adversarial loss
on its renderings. To ensure the latent code does not depend on 2D viewpoints
but truly reflects the 3D edits, we also regularize the latent code through a
contrastive learning scheme. Extensive experiments on various editing tasks
show GenN2N, as a universal framework, performs as well or better than
task-specific specialists while possessing flexible generative power. More
results on our project page: https://xiangyueliu.github.io/GenN2N/