Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal
  Rearrangement
Anthony Simeonov Ankit Goyal Lucas Manuelli Lin Yen-Chen Alina Sarmiento Alberto Rodriguez Pulkit Agrawal Dieter Fox
We propose a system for rearranging objects in a scene to achieve a desired
object-scene placing relationship, such as a book inserted in an open slot of a
bookshelf. The pipeline generalizes to novel geometries, poses, and layouts of
both scenes and objects, and is trained from demonstrations to operate directly
on 3D point clouds. Our system overcomes challenges associated with the
existence of many geometrically-similar rearrangement solutions for a given
scene. By leveraging an iterative pose de-noising training procedure, we can
fit multi-modal demonstration data and produce multi-modal outputs while
remaining precise and accurate. We also show the advantages of conditioning on
relevant local geometric features while ignoring irrelevant global structure
that harms both generalization and precision. We demonstrate our approach on
three distinct rearrangement tasks that require handling multi-modality and
generalization over object shape and pose in both simulation and the real
world. Project website, code, and videos:
https://anthonysimeonov.github.io/rpdiff-multi-modal/