Can Shape-Infused Joint Embeddings Improve Image-Conditioned 3D
  Diffusion?
Cristian Sbrolli Paolo Cudrano Matteo Matteucci
Recent advancements in deep generative models, particularly with the
application of CLIP (Contrastive Language Image Pretraining) to Denoising
Diffusion Probabilistic Models (DDPMs), have demonstrated remarkable
effectiveness in text to image generation. The well structured embedding space
of CLIP has also been extended to image to shape generation with DDPMs,
yielding notable results. Despite these successes, some fundamental questions
arise: Does CLIP ensure the best results in shape generation from images? Can
we leverage conditioning to bring explicit 3D knowledge into the generative
process and obtain better quality? This study introduces CISP (Contrastive
Image Shape Pre training), designed to enhance 3D shape synthesis guided by 2D
images. CISP aims to enrich the CLIP framework by aligning 2D images with 3D
shapes in a shared embedding space, specifically capturing 3D characteristics
potentially overlooked by CLIP's text image focus. Our comprehensive analysis
assesses CISP's guidance performance against CLIP guided models, focusing on
generation quality, diversity, and coherence of the produced shapes with the
conditioning image. We find that, while matching CLIP in generation quality and
diversity, CISP substantially improves coherence with input images,
underscoring the value of incorporating 3D knowledge into generative models.
These findings suggest a promising direction for advancing the synthesis of 3D
visual content by integrating multimodal systems with 3D representations.