Mosaic-SDF for 3D Generative Models
Lior Yariv Omri Puny Natalia Neverova Oran Gafni Yaron Lipman
Current diffusion or flow-based generative models for 3D shapes divide to
two: distilling pre-trained 2D image diffusion models, and training directly on
3D shapes. When training a diffusion or flow models on 3D shapes a crucial
design choice is the shape representation. An effective shape representation
needs to adhere three design principles: it should allow an efficient
conversion of large 3D datasets to the representation form; it should provide a
good tradeoff of approximation power versus number of parameters; and it should
have a simple tensorial form that is compatible with existing powerful neural
architectures. While standard 3D shape representations such as volumetric grids
and point clouds do not adhere to all these principles simultaneously, we
advocate in this paper a new representation that does. We introduce Mosaic-SDF
(M-SDF): a simple 3D shape representation that approximates the Signed Distance
Function (SDF) of a given shape by using a set of local grids spread near the
shape's boundary. The M-SDF representation is fast to compute for each shape
individually making it readily parallelizable; it is parameter efficient as it
only covers the space around the shape's boundary; and it has a simple matrix
form, compatible with Transformer-based architectures. We demonstrate the
efficacy of the M-SDF representation by using it to train a 3D generative flow
model including class-conditioned generation with the 3D Warehouse dataset, and
text-to-3D generation using a dataset of about 600k caption-shape pairs.