PyTorchGeoNodes: Enabling Differentiable Shape Programs for 3D Shape
  Reconstruction
Sinisa Stekovic Stefan Ainetter Mattia D'Urso Friedrich Fraundorfer Vincent Lepetit
We propose PyTorchGeoNodes, a differentiable module for reconstructing 3D
objects from images using interpretable shape programs. In comparison to
traditional CAD model retrieval methods, the use of shape programs for 3D
reconstruction allows for reasoning about the semantic properties of
reconstructed objects, editing, low memory footprint, etc. However, the
utilization of shape programs for 3D scene understanding has been largely
neglected in past works. As our main contribution, we enable gradient-based
optimization by introducing a module that translates shape programs designed in
Blender, for example, into efficient PyTorch code. We also provide a method
that relies on PyTorchGeoNodes and is inspired by Monte Carlo Tree Search
(MCTS) to jointly optimize discrete and continuous parameters of shape programs
and reconstruct 3D objects for input scenes. In our experiments, we apply our
algorithm to reconstruct 3D objects in the ScanNet dataset and evaluate our
results against CAD model retrieval-based reconstructions. Our experiments
indicate that our reconstructions match well the input scenes while enabling
semantic reasoning about reconstructed objects.