Guide3D: Create 3D Avatars from Text and Image Guidance
Yukang Cao Yan-Pei Cao Kai Han Ying Shan Kwan-Yee K. Wong
Recently, text-to-image generation has exhibited remarkable advancements,
with the ability to produce visually impressive results. In contrast,
text-to-3D generation has not yet reached a comparable level of quality.
Existing methods primarily rely on text-guided score distillation sampling
(SDS), and they encounter difficulties in transferring 2D attributes of the
generated images to 3D content. In this work, we aim to develop an effective 3D
generative model capable of synthesizing high-resolution textured meshes by
leveraging both textual and image information. To this end, we introduce
Guide3D, a zero-shot text-and-image-guided generative model for 3D avatar
generation based on diffusion models. Our model involves (1) generating
sparse-view images of a text-consistent character using diffusion models, and
(2) jointly optimizing multi-resolution differentiable marching tetrahedral
grids with pixel-aligned image features. We further propose a similarity-aware
feature fusion strategy for efficiently integrating features from different
views. Moreover, we introduce two novel training objectives as an alternative
to calculating SDS, significantly enhancing the optimization process. We
thoroughly evaluate the performance and components of our framework, which
outperforms the current state-of-the-art in producing topologically and
structurally correct geometry and high-resolution textures. Guide3D enables the
direct transfer of 2D-generated images to the 3D space. Our code will be made
publicly available.