Learning 3D Particle-based Simulators from RGB-D Videos
William F. Whitney Tatiana Lopez-Guevara Tobias Pfaff Yulia Rubanova Thomas Kipf Kimberly Stachenfeld Kelsey R. Allen
Realistic simulation is critical for applications ranging from robotics to
animation. Traditional analytic simulators sometimes struggle to capture
sufficiently realistic simulation which can lead to problems including the well
known "sim-to-real" gap in robotics. Learned simulators have emerged as an
alternative for better capturing real-world physical dynamics, but require
access to privileged ground truth physics information such as precise object
geometry or particle tracks. Here we propose a method for learning simulators
directly from observations. Visual Particle Dynamics (VPD) jointly learns a
latent particle-based representation of 3D scenes, a neural simulator of the
latent particle dynamics, and a renderer that can produce images of the scene
from arbitrary views. VPD learns end to end from posed RGB-D videos and does
not require access to privileged information. Unlike existing 2D video
prediction models, we show that VPD's 3D structure enables scene editing and
long-term predictions. These results pave the way for downstream applications
ranging from video editing to robotic planning.