Boosting3D: High-Fidelity Image-to-3D by Boosting 2D Diffusion Prior to
  3D Prior with Progressive Learning
Kai Yu Jinlin Liu Mengyang Feng Miaomiao Cui Xuansong Xie
We present Boosting3D, a multi-stage single image-to-3D generation method
that can robustly generate reasonable 3D objects in different data domains. The
point of this work is to solve the view consistency problem in single
image-guided 3D generation by modeling a reasonable geometric structure. For
this purpose, we propose to utilize better 3D prior to training the NeRF. More
specifically, we train an object-level LoRA for the target object using
original image and the rendering output of NeRF. And then we train the LoRA and
NeRF using a progressive training strategy. The LoRA and NeRF will boost each
other while training. After the progressive training, the LoRA learns the 3D
information of the generated object and eventually turns to an object-level 3D
prior. In the final stage, we extract the mesh from the trained NeRF and use
the trained LoRA to optimize the structure and appearance of the mesh. The
experiments demonstrate the effectiveness of the proposed method. Boosting3D
learns object-specific 3D prior which is beyond the ability of pre-trained
diffusion priors and achieves state-of-the-art performance in the single
image-to-3d generation task.