Progressive Learning of 3D Reconstruction Network from 2D GAN Data
Aysegul Dundar Jun Gao Andrew Tao Bryan Catanzaro
This paper presents a method to reconstruct high-quality textured 3D models
from single images. Current methods rely on datasets with expensive
annotations; multi-view images and their camera parameters. Our method relies
on GAN generated multi-view image datasets which have a negligible annotation
cost. However, they are not strictly multi-view consistent and sometimes GANs
output distorted images. This results in degraded reconstruction qualities. In
this work, to overcome these limitations of generated datasets, we have two
main contributions which lead us to achieve state-of-the-art results on
challenging objects: 1) A robust multi-stage learning scheme that gradually
relies more on the models own predictions when calculating losses, 2) A novel
adversarial learning pipeline with online pseudo-ground truth generations to
achieve fine details. Our work provides a bridge from 2D supervisions of GAN
models to 3D reconstruction models and removes the expensive annotation
efforts. We show significant improvements over previous methods whether they
were trained on GAN generated multi-view images or on real images with
expensive annotations. Please visit our web-page for 3D visuals:
https://research.nvidia.com/labs/adlr/progressive-3d-learning