Envision3D: One Image to 3D with Anchor Views Interpolation
Yatian Pang Tanghui Jia Yujun Shi Zhenyu Tang Junwu Zhang Xinhua Cheng Xing Zhou Francis E. H. Tay Li Yuan
We present Envision3D, a novel method for efficiently generating high-quality
3D content from a single image. Recent methods that extract 3D content from
multi-view images generated by diffusion models show great potential. However,
it is still challenging for diffusion models to generate dense multi-view
consistent images, which is crucial for the quality of 3D content extraction.
To address this issue, we propose a novel cascade diffusion framework, which
decomposes the challenging dense views generation task into two tractable
stages, namely anchor views generation and anchor views interpolation. In the
first stage, we train the image diffusion model to generate global consistent
anchor views conditioning on image-normal pairs. Subsequently, leveraging our
video diffusion model fine-tuned on consecutive multi-view images, we conduct
interpolation on the previous anchor views to generate extra dense views. This
framework yields dense, multi-view consistent images, providing comprehensive
3D information. To further enhance the overall generation quality, we introduce
a coarse-to-fine sampling strategy for the reconstruction algorithm to robustly
extract textured meshes from the generated dense images. Extensive experiments
demonstrate that our method is capable of generating high-quality 3D content in
terms of texture and geometry, surpassing previous image-to-3D baseline
methods.