Sketch and Text Guided Diffusion Model for Colored Point Cloud
  Generation
Zijie Wu Yaonan Wang Mingtao Feng He Xie Ajmal Mian
Diffusion probabilistic models have achieved remarkable success in text
guided image generation. However, generating 3D shapes is still challenging due
to the lack of sufficient data containing 3D models along with their
descriptions. Moreover, text based descriptions of 3D shapes are inherently
ambiguous and lack details. In this paper, we propose a sketch and text guided
probabilistic diffusion model for colored point cloud generation that
conditions the denoising process jointly with a hand drawn sketch of the object
and its textual description. We incrementally diffuse the point coordinates and
color values in a joint diffusion process to reach a Gaussian distribution.
Colored point cloud generation thus amounts to learning the reverse diffusion
process, conditioned by the sketch and text, to iteratively recover the desired
shape and color. Specifically, to learn effective sketch-text embedding, our
model adaptively aggregates the joint embedding of text prompt and the sketch
based on a capsule attention network. Our model uses staged diffusion to
generate the shape and then assign colors to different parts conditioned on the
appearance prompt while preserving precise shapes from the first stage. This
gives our model the flexibility to extend to multiple tasks, such as appearance
re-editing and part segmentation. Experimental results demonstrate that our
model outperforms recent state-of-the-art in point cloud generation.