InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions
Jiale Xu Xintao Wang Yan-Pei Cao Weihao Cheng Ying Shan Shenghua Gao
Enhancing AI systems to perform tasks following human instructions can
significantly boost productivity. In this paper, we present InstructP2P, an
end-to-end framework for 3D shape editing on point clouds, guided by high-level
textual instructions. InstructP2P extends the capabilities of existing methods
by synergizing the strengths of a text-conditioned point cloud diffusion model,
Point-E, and powerful language models, enabling color and geometry editing
using language instructions. To train InstructP2P, we introduce a new shape
editing dataset, constructed by integrating a shape segmentation dataset,
off-the-shelf shape programs, and diverse edit instructions generated by a
large language model, ChatGPT. Our proposed method allows for editing both
color and geometry of specific regions in a single forward pass, while leaving
other regions unaffected. In our experiments, InstructP2P shows generalization
capabilities, adapting to novel shape categories and instructions, despite
being trained on a limited amount of data.