2L3: Lifting Imperfect Generated 2D Images into Accurate 3D
Yizheng Chen Rengan Xie Qi Ye Sen Yang Zixuan Xie Tianxiao Chen Rong Li Yuchi Huo
Reconstructing 3D objects from a single image is an intriguing but
challenging problem. One promising solution is to utilize multi-view (MV) 3D
reconstruction to fuse generated MV images into consistent 3D objects. However,
the generated images usually suffer from inconsistent lighting, misaligned
geometry, and sparse views, leading to poor reconstruction quality. To cope
with these problems, we present a novel 3D reconstruction framework that
leverages intrinsic decomposition guidance, transient-mono prior guidance, and
view augmentation to cope with the three issues, respectively. Specifically, we
first leverage to decouple the shading information from the generated images to
reduce the impact of inconsistent lighting; then, we introduce mono prior with
view-dependent transient encoding to enhance the reconstructed normal; and
finally, we design a view augmentation fusion strategy that minimizes
pixel-level loss in generated sparse views and semantic loss in augmented
random views, resulting in view-consistent geometry and detailed textures. Our
approach, therefore, enables the integration of a pre-trained MV image
generator and a neural network-based volumetric signed distance function (SDF)
representation for a single image to 3D object reconstruction. We evaluate our
framework on various datasets and demonstrate its superior performance in both
quantitative and qualitative assessments, signifying a significant advancement
in 3D object reconstruction. Compared with the latest state-of-the-art method
Syncdreamer~\cite{liu2023syncdreamer}, we reduce the Chamfer Distance error by
about 36\% and improve PSNR by about 30\% .