CLIPXPlore: Coupled CLIP and Shape Spaces for 3D Shape Exploration
Jingyu Hu Ka-Hei Hui Zhengzhe liu Hao Zhang Chi-Wing Fu
This paper presents CLIPXPlore, a new framework that leverages a
vision-language model to guide the exploration of the 3D shape space. Many
recent methods have been developed to encode 3D shapes into a learned latent
shape space to enable generative design and modeling. Yet, existing methods
lack effective exploration mechanisms, despite the rich information. To this
end, we propose to leverage CLIP, a powerful pre-trained vision-language model,
to aid the shape-space exploration. Our idea is threefold. First, we couple the
CLIP and shape spaces by generating paired CLIP and shape codes through sketch
images and training a mapper network to connect the two spaces. Second, to
explore the space around a given shape, we formulate a co-optimization strategy
to search for the CLIP code that better matches the geometry of the shape.
Third, we design three exploration modes, binary-attribute-guided, text-guided,
and sketch-guided, to locate suitable exploration trajectories in shape space
and induce meaningful changes to the shape. We perform a series of experiments
to quantitatively and visually compare CLIPXPlore with different baselines in
each of the three exploration modes, showing that CLIPXPlore can produce many
meaningful exploration results that cannot be achieved by the existing
solutions.