Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis
Jonathon Luiten Georgios Kopanas Bastian Leibe Deva Ramanan
We present a method that simultaneously addresses the tasks of dynamic scene
novel-view synthesis and six degree-of-freedom (6-DOF) tracking of all dense
scene elements. We follow an analysis-by-synthesis framework, inspired by
recent work that models scenes as a collection of 3D Gaussians which are
optimized to reconstruct input images via differentiable rendering. To model
dynamic scenes, we allow Gaussians to move and rotate over time while enforcing
that they have persistent color, opacity, and size. By regularizing Gaussians'
motion and rotation with local-rigidity constraints, we show that our Dynamic
3D Gaussians correctly model the same area of physical space over time,
including the rotation of that space. Dense 6-DOF tracking and dynamic
reconstruction emerges naturally from persistent dynamic view synthesis,
without requiring any correspondence or flow as input. We demonstrate a large
number of downstream applications enabled by our representation, including
first-person view synthesis, dynamic compositional scene synthesis, and 4D
video editing.